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INTRODUCTION 

Development  of  mechanisms  to  aid  operators  of  complex  systems  has  a  long  and 
rich  history  (Rouse,  1991 ).  A  wide  variety  of  approaches  has  been  developed  for  aiding 
operators  in  problem  formulation/structuring,  probability  estimation  and  updating, 
selection  among  alternatives,  and  task  execution  and  monitoring.  In  'econt  years, 
aiding  has  evolved  to  include  decision  support  systems  and  intelligent  systems. 

Unfortunately,  despite  the  availability  of  a  growing  number  of  "proof  of  concept’ 
studies,  most  aiding  systems  have  been  unsuccessful.  They  have  been  developed,  but 
not  fielded.  Not  infrequently,  they  have  been  fielded,  but  not  used  (Klein,  1989). 

It  can  be  argued  that  this  unfortunate  result  is  due  to  unacceptable  rigidity  in  the 
design  of  aiding  systems  and/or  inflexibility  in  the  aiding  or  automation  philosophy 
underlying  such  systems  (Rouse,  1988).  A  typical  approach  to  aiding/automation  is  to 
computerize  everything  that  can  possibly  be  computerized,  and  make  whatever  is  left 
over  for  humans  as  easy  as  possible.  The  realization  that  this  approach  often  does  not 
work  has  led  to  the  concept  of  adaptive  aiding  systems,  whereby  the  nature  of  the 
aiding,  as  well  as  whether  or  not  the  aiding  is  used,  are  modified  or  adapted  to  changing 
characteristics  of  tasks  and/or  operators'  needs. 

The  adaptive  aiding  concept  attempts  to  address  several  concepts  in  the 
complex  task  domain.  For  example,  the  complex  task  environments  for  which  adaptive 
aiding  systems  are  suited  often  have  multiple  operational  elements  which  compete  for 
the  same  (limited)  operator  and  system  resources.  The  nature  of  these  environments 
also  requires  a  distinction  between  discrete  selection  and  continuous  control  tasks. 
Further,  there  are  often  dynamic  task  execution  priorities  for  which  the  operator  is 
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responsible  for  maintaining.  All  of  these  factors  must  be  taken  into  account  when 
designing  an  operator  aiding  system. 

Over  the  past  two  decades,  substantial  conceptual  development  and  a  variety  of 
experimental  efforts  have  proven  the  value  of  the  adaptive  aiding  concept  -  see  Rouse 
(1988)  for  a  review  of  this  work.  These  efforts  have  culminated  in  an  initial  high  level 
framework  for  designing  adaptive  aids.  The  initial  answers  to  these  questions  suggest  a 
preliminary  set  of  design  principles  to  guide  design  decisions  (Rouse,  1988, 1991). 

This  design  framework  is  structured  in  terms  of  a  set  of  six  questions: 

•  What  is  adapted  to? 

•  Who  does  the  adapting? 

•  When  does  adaptation  occur? 

•  What  methods  of  adaptation  apply? 

•  How  is  adaptation  done? 

•  What  is  the  nature  of  communication? 

The  framework  also  includes  alternative  answers  to  these  questions  and  is  discussed 
below. 


A  primary  difficulty  with  these  questions  and  alternative  answers  is  a  lack  of  data 
upon  which  to  base  choices  among  alternatives.  The  set  of  design  principles  currently 
available  are  helpful,  but  by  no  means  sufficient  to  cover  the  problem  domain  suggested 
by  the  six  design  questions.  So  the  primary  problem  is  we  have  a  suitably  defined 
problem  domain,  but  we  do  not  have  a  good  sense  of  viable  solutions  within  that 
domain. 
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Initially,  it  might  appear  that  the  obvious  solution  to  this  problem  is  collection  of 
the  requisite  data  to  "fill  in"  the  needed  set  of  principles.  Unfortunately,  the  experimental 
effort  necessary  to  provide  this  data  is  at  least  impractical,  and  possibly  unimaginable. 
The  complexity  of  the  situations  where  adaptive  aiding  is  of  particular  value  make  the 
set  of  potentially  relevant  data  much  too  large  to  imagine  collecting  it. 

An  alternative  approach  to  this  problem  is  adopted  in  this  study.  The  emphasis 
of  this  work  is  to  define  the  data  engineers  use  in  specifying  requirements  for  adaptive 
decision  aids.  In  other  words,  rather  than  attempt  to  compile  data,  as  well  as  the 
resulting  principles,  to  cover  all  possible  answers  to  the  design  questions,  the  focus  at 
this  time  should  be  on  elaborating  and  validating  solutions  the  designers  are  employing 
currently  to  determine  what  features  of  the  de  facto  adaptive  aid  design  process  are 
contributing  to  their  success.  Further,  the  definition  of  prospective  design  principles  and 
guidelines  should  pay  particular  attention  to  the  data  designers  rely  on  in  making  their 
design  decisions. 

In  this  paper,  we  report  the  results  of  pursuing  this  approach.  An  experimental 
investigation  was  performed  involving  designers  whose  task  was  to  produce 
specifications  for  adaptive  aiding  within  aircraft  mission  scenarios.  Statistical  methods 
were  used  to  identify  relationships  among  a  variety  of  decision  making  attributes  and 
designers'  specification  decisions. 

BACKGROUND 

This  section  organizes  many  of  the  results  and  ideas  discussed  earlier.  In 
addition,  several  adaptive  aiding  notions  are  discussed  that  have  yet  to  receive  serious 
study.  A  primary  goal  of  this  section  is  to  illustrate  the  breadth  of  possibilities  when 
designing  adaptive  aids.  In  this  way  it  is  hoped  that  one  might  possibly  counteract  the 
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previously  mentioned  tendency  to  pursue  design  of  intelligent  support  systems  in  an  ad 
hoc  manner. 

We  have  found  that  the  conceptual  design  of  an  adaptive  aid  can  be  approached 
systematically  by  pursuing  answers  to  a  specific  set  of  design  questions  or  issues. 
While  it  is  not  possible  to  provide  generic,  context-free  answers  to  these  questions,  it  is 
possible  to  outline  the  range  of  alternative  answers  and  suggest  principles  of  adaptation 
and  interaction  that  may  assist  designers  in  choosing  among  these  alternatives. 

Design  Issues 

In  developing  a  framework  for  research  on  adaptive  aiding  several  years  ago 
(Rouse  and  Rouse,  1983),  a  structured  set  of  design  issues  emerged.  A  subset  of 
these  issues  is  quite  similar  to  the  framework  proposed  recently  by  Lehner  and  his 
colleagues  (1987).  The  overall  set  of  issues  is  discussed  in  this  section. 

What  is  adapted  to? 

In  general,  adaptation  to  the  user  and/or  task  is  possible.  In  addition,  an  aid  can 
adapt  to  a  class  of  users  (or  tasks),  a  particular  user  (or  task),  or  a  particular  user  (or 
task)  at  a  specific  point  in  time.  In  other  words,  adaptation  can  be  relative  to  a  class  as 
a  whole,  a  member  of  a  class,  or  the  state  of  a  particular  member. 

An  interesting  aspect  of  answering  this  question  concerns  whether  the  emphasis 
should  be  on  adapting  to  the  user  or  adapting  ofthe  user.  Although  it  is  often  the  case 
that  users’  needs  and  preferences  should  be  accommodated,  there  are  also  situations  in 
which  overall  performance  can  be  enhanced  by  providing  users  with  new  skills  and 
knowledge.  Current  interests  and  developments  in  embedded  training  and  intelligent 
tutoring  systems  are  providing  a  strong  basis  for  pursuing  the  notion  of  adapting  the 
user  as  well  as  the  aid.  Lehner  and  his  co-workers  (1987)  and  Noah  and  Hatpin  (1986) 
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have  advocated  embedded  training  as  a  potential  component  of  an  adaptive  aiding 
system. 

Who  does  the  adapting? 

If  viewed  very  narrowly,  this  question  has  only  one  possible  answer  -  the  aid 
adapts.  However,  from  a  broader  perspective  the  agent  of  adaptation  can  be  the 
system  designer,  users,  or  the  aid.  it  can  reasonably  be  argued  that  a  system  designer 
always  adapts  a  system  to  a  class  of  users  and  tasks.  Beyond  such  "static"  adaptation, 
users  often  configure  a  system  for  themselves  by,  for  example,  adjusting  the  seat  and 
mirrors,  choosing  autopilot  modes,  or  requesting  particular  report  formats.  The  aid  is 
the  appropriate  agent  of  adaptation  if  design  adaptations  need  to  be  refined  and/or 
changed  and  if  users  are  unlikely  to  perceive  the  need  or  be  able  to  execute  these 
adaptations. 

When  does  adaptation  occur? 

It  may  be  possible  to  adapt  off-line  prior  to  operation.  Alternatively,  adaptation 
can  occur  on-line  in  anticipation  of  changing  demands.  Finally,  adaptation  can  occur 
on-line  in  response  to  changes.  Clearly  the  "what,"  "who,"  and  "when"  questions 
interact  in  the  sense  that  not  all  possible  combinations  of  answers  are  feasible.  For 
example,  it  is  difficult  to  imagine  a  designer  being  the  agent  of  on-line  adaptation  to  the 
time-varying  state  of  a  particular  user. 

What  methods  of  adaptation  apply? 

There  are  three  general  methods  for  aiding  a  user:  (1)  an  aid  can  make  a  task 
easier,  (2)  an  aid  can  perform  part  of  a  task,  and  (3)  an  aid  can  completely  perform  a 
task.  These  three  methods  can  be  termed  transformation,  partitioning,  and  allocation, 
respectively  (Rouse  and  Rouse,  1983).  As  examples,  many  display  enhancement 
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techniques  (e.g.,  filtering  and  smoothing)  represent  transformations.  Display 
highlighting  (e.g.,  for  cautions,  alarms,  and  warnings)  is  an  example  cf  partitioning. 
Autopilots  represent  aiding  via  allocation. 

Although  the  distinctions  among  these  three  methods  of  adaptation  are  certainly 
not  crisp,  it  is  a  reasonably  straightforward  matter  to  decide  which  method  applies  if  one 
focuses  on  the  implications  for  the  role  of  users.  With  transformation,  users  still  perform 
the  task  in  question.  With  partitioning,  users  are  still  "in  the  loop”  but  are  not  the  sole 
agents  of  action.  With  allocation,  the  computer  is  the  only  active  agent  for  the  task  that 
has  been  allocated. 

How  is  adaptation  done? 

This  question  does  not  concern  how  the  aid  performs  a  portion  or  all  of  a  task. 
Rather,  the  issues  is  the  basis  for  determining  the  need  for  transformation,  partitioning, 
or  allocation.  There  are  basically  two  approaches  to  making  this  determination.  One 
approach  is  measurement,  directly  in  terms  of  performance  decrements  or  changes  of 
demands  or  indirectly  in  terms  of,  for  example,  leading  indicators  of  performance.  In  the 
study  conducted  by  Morris,  Rouse,  and  Frey  (1985),  it  was  found  that  the  average  time 
to  detect  a  target  once  it  came  into  view,  began  increasing  about  20  seconds  prior  to 
any  targets  being  missed.  This  detection  latency  served  as  a  leading  indicator  of  the 
primary  measure  of  detection  accuracy. 

The  other  approach  is  modeling,  whereby  predictions  of  intentions,  resource 
availability,  and  performance  can  be  used  to  trigger  adaptation.  This  approach  basically 
provides  the  agent  of  adaptation  with  "expectations”  the  violation  of  which  results  in  at 
least  more  targeted  monitoring  and  eventually  some  degree  of  adaptation.  Approaches 
based  on  mathematical  models  of  human  decision  making  (Reevesman  and 
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Greenstein.  1986),  and  Geddes's  model  for  intent  inferencing  (Geddes,  1990)  are  good 
examples  of  models  that  are  used  in  this  way. 

It  is  important  to  emphasize  that  depending  on  the  agent  of  adaptation  (e.g.,  user 
as  opposed  to  aid),  measurement  and  modeling  have  to  be  handled  differently.  If  the 
user  is  to  make  the  necessary  measurements,  the  requisite  information  must  be 
available  and  displayed  appropriately.  Further,  if  the  user  is  to  have  the  necessary 
"mental  models,"  it  is  likely  that  specialized  training  will  have  to  be  developed  (and 
perhaps  embedded)  as  well  as  associated  displays  for  supporting  the  use  of  these 
models.  In  contrast,  if  the  aid  is  the  agent  of  adaptation,  measurement  and  modeling 
must  focus  on  instrumentation  and  processing  issues  rather  than  displays  and  training. 

What  is  the  nature  of  communication? 

The  basic  issue  here  concerns  whether  communications  about  adaptation  should 
be  explicit  or  implicit.  Explicit  communication  between  user  and  aid  concerning  the 
activities,  awareness,  and  intentions  of  each  party  has  the  advantage  of  being  minimally 
ambiguous  but  can  impose  substantial  overhead.  The  cost  of  this  overhead  can 
potentially  exceed  the  benefits  of  aiding. 

In  contrast,  implicit  communication,  via  measurements  and/or  models,  can 
greatly  lessen  this  overhead  but  suffers  from  greater  uncertainty  and  ambiguity 
regarding  the  actions  and  intentions  of  each  party.  Revesman  and  Greenstein's  work 
(1986)  clearly  illustrates  the  trade-off  in  choosing  between  explicit  and  implicit 
communication.  The  trade-off  hinges  on  the  uncertainty  associated  with  model-based 
implicit  communication.  If  a  model  can  provide  perfect  predictions  of  the  user’s 
intentions  and  actions,  there  is  no  need  to  communicate  explicitly,  and  thus  the  cost  of 
explicit  communication  can  be  avoided.  However,  as  uncertainty  grows,  predictions  will 
more  frequently  be  wrong,  and  as  a  result,  tasks  will  slip  through  \he  cracks  or  receive 
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redundant  efforts.  To  avoid  these  possibilities,  increased  explicit  communication  is 
needed  to  check  or  calibrate  a  model’s  predictions.  If  the  level  of  uncertainty  associated 
with  a  model’s  predictions  becomes  too  great,  totally  explicit  communications  become 
the  best  policy. 

Principles  of  Adaptation 

The  current  data  base  of  empirical  studies  of  adaptive  aiding  is  much  too  meager 
to  codify  a  principia  adaptivia  (Rouse,  1988).  However,  sufficient  R&D  experience  has 
been  gained  to  be  able  to  outline  the  general  nature  of  the  requisite  principles  and 
suggest,  at  least  tentatively,  specific  design  principles. 


Two  types  of  principles  are  needed:  (1)  principles  of  adaptation  and  (2) 
principles  of  interaction.  Principles  of  adaptation  concern  when  and  how  adaptive  aiding 
applies,  as  well  as  the  underlying  mechanisms  of  adaptation.  The  experimental  results 
discussed  earlier,  as  well  as  some  recent  recommendations  by  other  researchers, 
suggest  the  following  incomplete  set  of  principles: 

•  The  need  for  aiding  can  depend  on  the  interaction  of  impending  and 
recently  completed  task  demands  (Morris  and  Rouse.  1986)  -  task 
allocation  decisions  should  not  be  based  solely  on  the  demands  of  the 
task  in  question. 

•  The  availability  of  aiding  and  who  does  the  adapting  can  affect 
performance  when  the  aid  is  not  in  use  (Morris  and  Rouse,  1986)  -- 
total  system  performance  may  be  enhanced  by  keeping  the  user  in 
charge  of  allocation  decisions. 

•  When  using  measurements  as  a  basis  for  adaptation,  temporal  patterns 
of  user  and  system  behavior  can  provide  leading  indicators  of  needs  for 
aiding  (Morris,  Rouse,  and  Frey,  1985:  Morris,  House,  Ward,  and  Frey, 

1984)  --  it  may  be  possible  to  use  secondary  indices  as  proxy  measures 
of  the  indices  of  primary  concern. 

•  When  using  models  as  a  basis  for  adaptation,  the  degree  of  task 
structure  will  dictate  the  accuracy  with  which  inferences  of  activities, 
awareness,  and  intentions  can  be  made  (Rouse,  Geddes,  and  Curry, 

1986,  1987)  --  tasks  with  substantial  levels  of  user  discretion  may  limit 
the  potential  of  model-based  adaptation. 
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To  the  extent  possible,  incorporate  within  the  aid  models  that  allow 
predictions  of  the  relative  abilities  of  users  and  the  aid  to  perform  the 
task  in  particular  situations  (Lehner  et  al.,  1987;  Morris  and  Rouse, 
1986)  -  substantial  variations  of  relative  abilities  of  users  and  aids 
provide  the  central  impetus  for  adaptation. 


Although  the  foregoing  principles  are  qualitative,  they  help  to  answer  some  of  the 
design  questions  posed  earlier.  For  example,  answering  the  question  of  who  does  the 
adapting  can  be  seen  to  depend  on  task  structure,  likely  task  sequences,  and  the  extent 
to  which  appropriate  measurement  and  modeling  methods  are  available  and 
computationally  viable. 


Principles  of  Interaction 


These  principles  relate  to  the  characteristics  of  adaptive  aiding  that  foster  (or 
hinder)  humans'  acceptance  and  utilization  of  these  aids.  Principles  of  interaction  also 
concern  the  extent  to  which  users  must  understand  the  functioning  of  adaptive  aiding  in 
order  to  utilize  the  aiding  appropriately  and  determine  whether  or  not  it  is  functioning 
properly.  Finally,  of  course,  these  principles  relate  to  the  somewhat  traditional  hurna" 
factors  issues  of  the  displays  and  controls  associated  explicitly  with  adaptive  aiding. 
Based  on  the  experimental  results  summarized  earlier  and  various  researchers’ 
recommendations,  the  following  incomplete  set  of  principles  is  suggested: 

•  Users  can  perceive  themselves  as  performing  better  than  they  actually 
do  and  may  want  an  aid  to  be  better  than  they  are  (Morris  and  Rouse, 

1986)  -*  as  a  result,  an  aid  may  have  to  be  much  better  than  users  in 
order  to  be  accepted. 

•  Ensure  that  user-initiated  adaptation  is  possible  and  appropriately 
supported,  even  if  aid-initiated  adaptation  is  the  norm  (Lehner  et  al., 

1987;  Noah  and  Halpin,  1986)  --  ensure  that  users  feel  they  are  in 
charge  even  if  they  have  delegated  authority  to  the  aid. 

•  Provide  means  to  avoid  user  confusion  in  reaction  to  aid-initiated 
adaptation  (Lehner,  et  al.,  1987)  and  methods  for  the  user  to  preempt 
adaptation  (Chu  and  Rouse,  1979)  --  make  it  very  clear  whether  human 
or  computer  is  supposed  to  perform  a  particular  task  at  a  specific  time 
and  provide  means  for  changing  this  allocation. 
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It  appears  that  aid-initiated  "off-loading"  oi  the  user  and  user-initiated 
recapturing  of  tasks  is  a  viable  means  of  avoiding  "hot  potato*  trading  of 
task  responsibilities  (Rouse,  Geddes,  and  Curry,  1986,  1987)  --  this 
asymmetry  may  help  to  ensure  that  users  will  feel  in  charge  of  the 
overall  system. 

There  is  a  trade-off  between  the  predictive  abilities  (i.e.,  in  terms  of 
uncertainty  reduction)  of  models  of  human  performance  and  intent  and 
the  way  in  which  the  explicit  versus  implicit  communication  issue  is 
resolved  (Revesman  and  Greenstein,  1986)  ~  the  cost  of  explicit 
communication  (e.g.,  workload  and  time  required)  should  be  compared 
with  the  cost  of  adaptation  errors  (i.e.,  misses  and  false  alarms). 

The  extent  to  which  users  can  be  appropriate  agents  of  adaptation  may 
depend  on  their  models  of  the  functioning  of  the  aid  and  themselves 
(Lehner,  et  al.,  1987;  Morris  and  Rouse,  1986;  Morris.  Rouse,  and 
Ward,  1985)  --  adaptation  of  the  user  (e.g.,  via  embedded  training)  is  a 
viable  approach  for  providing  such  models. 

A  variety  of  specific  human  factors  principles  for  design  of  complex 
information  systems  appear  to  apply  to  the  design  of  the  displays  and 
controls  associated  with  adaptive  aiding  -  see  the  list  provided  by  Noah 
and  Halpin  (1986). 


The  design  questions  discussed  in  this  section,  as  well  as  the  tentative  and 
incomplete  list  of  principles  of  adaptation  and  interaction,  illustrate  the  maturity  of  the 
concept  of  adaptive  aiding.  A  variety  of  researchers  have  invested  substantial  efforts  in 
the  area,  and  as  a  result,  the  concept  has  moved  substantially  beyond  the  ad  hoc  status 
of  much  decision-aiding  technology.  Nevertheless,  there  is  a  surprising  paucity  of  data 
upon  which  to  base  firm  conclusions.  Much  of  what  has  been  outlined  and  suggested  in 
this  section  merits  efforts  aimed  at  replication  and  generalization  of  results  and 
interpretations. 

Aiding  Scenarios 

Using  the  framework  discussed  in  the  previous  section  as  a  starting  point,  we 
began  to  investigate  designers'  decision  making  processes.  In  order  to  replicate  the 
design  environment  as  closely  as  possible,  we  provided  the  designers  with  typical  aiding 
design  context:  the  target  system  functional  scenario.  Functional  scenarios  are  oftnn  a 
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major  source  of  information  for  top-level  designers,  and  are  used  by  designers  to 
determine  how  specific  aiding  functions  will  be  realized  in  the  target  system. 


The  task  to  be  addressed  utilized  a  mission  scenario  for  a  2000+  fighter  aircraft 
involved  in  a  beyond-visual-range  attack  engagement.  This  seven-page  scenario  was 
decomposed  into  42  scenario  events,  each  characterized  as  shown  in  Figure  1.  The 
four  elements  of  the  event  descriptions  are  shown  in  Figure  1  and  described  below. 


1  sa:  Information  seeking  | 

CROWN  provides  as  much  targeting  information  as  possible  as  the  two 
orces  close  to  about  1 50  miles.  This  information  is  transmitted  to  the  Blue 
Wight's  aircraft  fire  control  systems  via  secure  link  where  it  appears  on  each 
aircraft's  tactical  situation  display. 

6.0  Intercept 

6.33  Correlate  external  data  with  on-board  data/information 

6.42  Perform  target  acquisition 

6.43  Perform  target  ID 

6.44  Assess  raid 

6.45  Determine  target  assignments 

6.46  Determine  preliminary  targeting 

1  1 

6.15  Maintain  formation/mutual  support 

_ 

Figure  1 .  Scenario  Event  Example 


First,  the  general  user-system  task  is  shown.  In  this  case,  the  task  was  judged  to 
be  situation  assessment  ((SA):  information  seeking).  Events  were  characterized  in  this 
manner  using  Rouse's  task  taxonomy.  This  taxonomy  (Figure  2)  has  been  found  to  be 
useful  in  a  variety  of  efforts  involving  design  of  aiding  systems  for  command  and  control, 
nuclear  power,  manufacturing,  and  design  information  systems  (Rouse,  1986,  1991). 
For  the  purposes  of  this  study,  only  the  main  four  categories  were  employed: 
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•  Execution  and  monitoring, 

•  Situation  assessment:  information  seeking, 

•  Situation  assessment:  explanation,  and 

•  Planning  and  commitment. 

Execution  and  Monitoring 

1 .  Implementation  of  Plan 

2.  Observation  of  Consequences 

3.  Evaluation  of  Deviations  from  Expectations 

4.  Selection  Between  Acceptance  and  Rejection 

Situation  Assessment:  Information  Seeking 

5.  Generation/Identification  of  Alternative  Information  Sources 

6.  Evaluation  of  Alternative  Information  Sources 

7.  Selection  Among  Alternative  Information  Sources 

Situation  Assessment:  Explanation 

8.  Generation  of  Alternative  Explanations 

9.  Evaluation  of  Alternative  Explanations 

10.  Selection  Among  Alternative  Explanations 

Planning  and  Commitment 

1 1 .  Generation  of  Alternative  Courses  of  Action 

12.  Evaluation  of  Alternative  Courses  of  Action 

1 3.  Selection  Among  Alternative  Courses  of  Action 

Figure  2.  Taxonomy  of  User-System  Tasks 

Each  of  the  42  scenario  events  were  classified  by  two  independent  analysts  into 
one  of  these  categories.  The  small  percentage  of  disagreements  were  resolved  by 
discussing  the  elements  of  the  event  in  question  and  reaching  a  consensus  on  its 
classification. 
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The  second  element  of  Figure  1  is  a  prose  description  of  the  event.  This 
information  provides  context,  as  well  as  mission-related  links  to  the  rest  of  the  scenario. 
This  context  is  critical  to  designers  being  able  to  relate  to  the  design  task  that  they  were 
being  asked  to  do. 

The  third  element,  shown  within  a  single  box,  describes  the  foreground  tasks  for 
which  aiding  might  be  specified.  This  information  is  characterized  using  Cohen's 
taxonomy  for  advanced  aircraft  operations  (Cohen,  1990).  This  characterization 
assured  that  all  designers  were  given  the  same  task  requirements. 

The  fourth  and  final  element  of  Figure  1,  shown  within  a  double  box,  describes 
the  background  tasks  that  must  be  performed  despite  the  emergence  of  new  foreground 
demands.  The  distinction  between  foreground  and  background  tasks  provides 
designers  with  the  possibility  of  aiding  new  demands  and/or  ongoing  demands.  This 
distinction  is  important  because  new  demands  can  be  satisfied  by  either  aiding  these 
demands,  or  by  aiding  other  tasks,  thereby  freeing  the  operator  to  address  the  new 
demands. 

The  complete  description  of  all  42  scenario  events,  as  well  as  the  decomposition 
process  used  to  classify  and  characterize  events,  is  provided  in  Appendix  A.  The 
appendix  also  describes  in  much  more  detail,  the  experiment,  data  analysis,  and  results 
presented  in  the  remainder  of  this  paper. 

Specification  of  Aiding 

For  each  of  the  42  scenario  events,  designers  were  asked  to  respond  to  the 
multiple  choice  questions  shown  in  Figure  3.  For  the  "motivation"  category,  designers 
were  asked  to  rank  order  the  four  alternatives  from  most  important  reason  for  aiding  to 
least  important. 
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Motivation  for  Specifying  Adaptive  Aiding 

-  Estimated  performance  degradation  without  aiding 

-  Projected  workload  in  the  scenario  event 

-  Tactical  significance  of  scenario  event 

-  Projected  implementation  practicality 

Tasks  to  be  Aided 

-  Foreground 

-  Background 

Type  of  Adaptive  Aiding 

-  Allocation 

-  Partitioning 

-  Transformation 

•  None 

Method  of  Aid  Invocation 

•  Unacceptable  system  performance 

-  Number  of  concurrent  operator  tasks 

-  Operator  resource  allocation  estimate 

•  Nature  of  task  (Fitts’  list  reference) 

-  Other  (e.g.,  operator  requests  aiding) 

Operator-Aid  Communication  Requirements 

-  Procedural 

-  Product 

-  Process 


Figure  3.  Specification  Categories 


Designers  could  respond  to  the  "tasks  to  be  aided"  category  by  specifying 
neither,  either,  or  both  foreground  and  background  tasks.  As  noted  earlier,  the  choice 
here  concerns  aiding  new  demands  or  aiding  existing  demands  to  enable  reallocation  of 
attention  to  new  demands.  If  designers  specified  both  foreground  and  background 
tasks,  then  two  specification  sheets  were  filled  out  for  the  event,  one  for  foreground  and 
one  for  background  tasks. 
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The  "type  of  adaptive  aiding"  category  in  Figure  3  included  four  possible 
responses  --  three  types  of  aiding  and  a  fourth  choice  of  no  aiding  at  all.  These  three 
types  of  aiding  are  those  postulated  in  the  adaptive  aiding  design  framework  (Rouse, 
1988). 


For  allocation,  the  aiding  system  assigns  task  execution  activities  to  itself. 
Operator  coordination  of  task  performance  is  not  necessary.  While  the  operator  must 
be  notified  of  allocation  recommendations/decisions,  once  the  aid  is  activated,  it  carries 
execution  to  completion  unless  operator  decides  to  resume  task  performance. 

With  partitioning,  operator  and  aid  "share"  task  execution.  In  most  cases,  the  aid 
will  indicate  what  it  can  do  (e.g.,  target  designation)  while  the  operator  retains  remaining 
portions  of  tasks  (e.g.,  target  identification).  Aspects  of  the  task  (sub-tasks)  are  shifted 
between  agents.  Partitioning  of  tasks  requires  that  operator  and  aid  share  information 
to  coordinate  task  performance. 

Transformation  involves  modifying  a  (possibly  increasingly)  difficult  task  to 
mitigate  task  demands.  For  example,  an  operator  engaged  in  a  demanding  flight  control 
task  in  conjunction  with  a  difficult  situation  assessment  task  (e.g.,  due  to  subsystem 
failure)  might  be  aided  by  transforming  flight  control  displays  to  allow  a  simpler  mode  of 
tracking.  Conceptually,  the  requirements  for  performing  the  task  are  changed  by  the 
aiding,  but  the  pilot  remains  involved  with  the  task. 

The  "method  of  aid  invocation"  category  in  Figure  3  relates  to  the  intervention 
"threshold"  used  to  activate  aiding.  The  alternative  responses  in  this  category  are 
reasonably  self  explanatory,  with  the  possible  exception  of  the  reference  to  Fitts’  list. 
This  refers  to  the  classic  "men  are  better  at/machines  are  better  at"  lists  that  Fitts 
originated  (Fitts,  1951).  Several  alternative  lists  of  this  type  are  currently  available. 
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The  final  category  in  Figure  3,  "operator-aid  communication  requirements,"  refers 
to  the  types  of  information  that  operator  and  aid  can  share.  The  three  types  of 
information  include: 

*  Procedural  information  -  A  primary  type  of  aid  status  information. 

Refers  to  information  pertaining  to  when  to  use  the  aid,  or  for 
determining  intervention  thresholds  (Morris,  Rouse,  and  Ward,  1985). 

♦  Process  information  -  A  primary  type  of  aid  status  information.  Refers 
to  functional  information  about  the  aid;  information  about  the  process 
by  which  the  aid  accomplishes  its  tasks.  This  information  may  allow  the 
operator  to  determine  tne  applicability  of  the  aid  to  the  current  situation 
(Morris,  Rouse,  and  Ward,  1985). 

•  Product  information  -  A  primary  type  of  aid  status  information.  Refers  to 
information  about  normal  aiding  system  output  that  allows  the  operator 
to  determine  whether  the  system  is  functioning  properly  (Morris,  Rouse, 
and  Ward,  1985). 


Subjects  were  asked  to  respond  to  th'~  category  by  rating  (0-10)  the  relative 
amount  of  information  needed  of  each  type.  These  three  types  of  information  were 
chosen  based  on  an  analysis  of  information  requirements  for  adaptive  aiding  (Morris, 
Rouse,  &  Ward,  1985).  The  results  of  this  analysis  indicated  that  human  interaction  with 
adaptive  aiding  systems  is  likely  to  be  substantially  affected  by  the  extent  to  which 
procedural,  product,  and  process  information  is  available. 


Decision  Making  Attributes 


In  addition  to  specifying  adaptive  aiding  using  the  categories  in  Figure  3,  subjects 
were  asked  to  rate  (0-10)  the  twelve  attributes  listed  in  Figure  4.  The  purpose  of  these 
ratings  was  to  assess  the  characteristics  of  the  aiding  situation  that  appeared  to  relate 
to  specification  decisions.  Attribute  ratings  were  performed  for  each  scenario  event 
subsequent  to  completion  of  the  aiding  specification  sheet  for  that  event.  Subjects  were 
asked  to  rate  the  importance  of  each  attribute  to  the  eventual  success  of  the  specified 
aid  (0  =  not  at  all,  10  =  critical). 
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The  twelve  attributes  in  Figure  4  were  defined  as  follows: 

1 .  Anticipated  Aiding  Intervention  Criterion  -  One  of  the  principle  design 
questions  that  the  designer  must  face  is  whether  or  not  to  aid  the 
operator.  Success  of  the  aiding  system  will  greatly  depend  on  what 
criterion  is  used  in  answering:  *Under  what  circumstances  should  the 
aid  intervene?"  There  are  several  criteria  upon  which  aid  intervention 
can  be  based  (e.g.,  unacceptable  operator  performance,  number  of 
concurrent  tasks,  operator  errors).  The  criterion  must  be  considered  in 
the  context  of  aiding.  Within  this  context,  the  designer  must  also 
consider  the  anticipated  knowledge  representation  of  the  supporting 
architecture. 

2.  Tradeoffs  between  cost  of  communication  with  the  operator  about  error 
vs.  aiding  -  In  specifying  aiding  to  assist  the  operator,  for  example, 
when  he  commits  critical  (i.e.,  life  threatening)  errors,  the  designer 
should  consider  several  factors  (e.g.,  time  pressure,  severity  of  error, 
intervention  criterion,  etc.)  in  deciding  whether  to  communicate  with  the 
operator  about  an  error  or  immediately  activate  aiding  to  compensate 
for  the  error. 

3.  Anticipated  difficulty  of  implementing  the  aid  -  Deciding  whether  or  not 
to  aid  the  operator  is  often  influenced  by  how  difficult  the 
implementation  of  such  a  system  may  be.  Additionally,  the  type  of 
aiding  and  interaction  with  the  operator  will  be  affected  by  this 
consideration.  This  attribute  should  be  considered  in  terms  of  the  level 
of  aid  functionality  and  level  of  technology  embedded  in  the  aiding 
system. 

4.  Anticipated  reliability  of  aid  behavior  in  normal  vs.  novel  situations  -  An 
aiding  system  is  only  as  effective  as  designed.  In  this  context,  reliability 
is  defined  as  the  expected,  repeatable  performance  of  the  aid,  not 
mean  time  between  failures  of  the  aid.  The  behavioral  science 
definition  for  reliability  is  used  here  instead  of  the  engineering  definition. 
We  are  more  concerned  with  the  expected  vs.  actual  behavior  of  the  aid 
in  novel  error  situations.  In  other  words,  can  the  operator  rely  on  the 
aid's  functionality  in  novel  situations? 

5.  Necessary  types  and  level  of  detail  of  operator-aid  communication  -  In 
order  to  facilitate  elective  coordination  between  the  aid  and  the 
operator,  the  aid  must  communicate  useful  information  to  the  operator. 
The  operator  can  receive  information  about  what  the  aid  is  doing 
(procedural),  what  the  aid’s  outputs  are  (product),  or  how  the  aid  is 
executing  the  task  (process).  The  designer  should  consider  what 
information  requirements  the  operator  will  nave  about  the  aid  and  the 
necessary  detail  of  that  information. 
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1 .  Anticipated  aiding  intervention  criterion 

2.  Tradeoffs  between  costs  of  communicating  with  operator 
about  error  vs.  aiding 

3.  Anticipated  difficulty  of  implementing  the  aid 

4.  Anticipated  reliability  of  aid  behavior  in  norma!  vs.  novel 
situations 

5.  Necessary  types  and  level  of  detail  of  operator-aid 
communication 

6.  Overall  risk  (from  design  perspective)  of  aiding  this  event 

7.  Anticipated  ease  of  aiding  introduction  and  removal 

8.  Suspected  user  attitude  towards  aiding 

9.  Essential  information  requirements  for  effective  aiding 

1 0.  Necessary  level  of  aid  tailorability 

1 1 .  Availability  of  technology  to  support  aiding  implementation 

1 2.  Number  and  applicability  of  interface/aiding  models 
available 


Figure  4.  Decision  Making  Attributes 

6.  Overall  risk  (from  design  perspective)  of  aiding  an  event  *  The  overall 
risk  rating  is  of  paramount  importance  to  the  specification  of  an 
adaptive  aiding  system.  Risk  is  defined  as  what  the  designer  is  willing 
to  trade  off  for  potentially  high  functionality.  For  example,  specifying  an 
aid  that  will  intervene  in  critical  error  situations  and  save  the  operator’s 
life,  albeit  through  unpredictable  behavior,  may  be  worth  the  interaction 
risk. 

7.  Anticipated  ease  of  aiding  introduction  and  removal  -  The  designer  must 
consider  the  ease  with  which  aiding  can  be  introduced  into  the  task 
environment.  For  example,  will  the  operator  perceive  a  lack  of 
"cognitive  unity"  when  a  task  transformation  is  introduced?  It  is  also 
important  that  the  negative  cognitive  and  perceptual  effects  of  removal 
of  adaptive  aiding  be  minimized,  in  this  case,  the  designer  must 
consider  the  costs  of  removal  of  aiding  vs.  the  benefits  of  allowing  the 
aid  to  execute  a  task  to  completion. 

8.  Suspected  user  attitude  towards  aiding  -  Some  types  of  aiding  are  more 
acceptable  to  a  user  population  than  others.  If  the  designer  is 
specifying  a  risky  adaptive  aiding  system  from  the  operator’s  point  of 
view,  for  example,  the  designer  should  consider  whether  the  operator 
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will  want  to  use  it.  The  operator  must  be  (or  become)  comfortable  with 
an  aiding  system  before  he  will  use  it. 

9.  Essential  information  requirements  for  effective  aiding  -  The  information 
requirements,  necessary  to  facilitate  the  aiding  process,  are  important 
considerations  in  specifying  aiding.  Information  requirements  for  the 
operator  about  the  aid,  as  well  as  information  for  the  aid  about  the 
operator,  will  determine  how  and  what  will  be  aided  in  the  system. 

1 0.  Necessary  level  of  su'd  tailorability  -  How  much  of  the  aid's  behavior  can 
(and  should)  be  tailored  based  on  individual  differences  and/or 
population  differences?  This  attribute  affects  aiding  intervention 
thresholds  (e.g.,  "What  is  the  value  that  determines  unacceptable 
performance  for  this  operator?",  etc.).  In  addition,  this  could  pertain  to 
the  level  of  communication  between  the  operator  and  aid  within  a 
particular  task  context). 

1 1 .  Available  technology  to  support  aiding  implementation  -  Even  though 
we  are  analyzing  scenarios  for  future  aircraft,  the  designer  must 
consider  what  role  technology  push  and/or  puli  will  play  in  implementing 
some  adaptive  aiding  systems.  Consider  the  range  to  be  from  none  (all 
technology  must  be  developed  to  support  this  design)  to  all  technology 
available  now  "off-the-shelf." 

12.  Number  and  applicability  of  interface/aiding  models  available  -  Tools, 
task  models,  simulations,  etc.  allow  the  designer  to  gain  insight  into  the 
process  that  he  wishes  to  aid.  In  addition,  embeddable  models  may 
facilitate  better  interaction  and  aid  functionality.  When  specifying  the 
aiding  system,  consider  the  number  of  available  models,  their 
applicability,  and  anticipated  success  of  using  such  models. 


The  above  definitions  were  provided  to  subjects  prior  to  beginning  the  aiding 
specification  process  and  were  available  for  reference  throughout  the  experiment. 


The  analysis  whereby  the  above  attributes  were  identified  is  presented  in 
Appendix  B  of  this  report.  Basically,  this  analysis  process  involved  reviewing  a  wide 
range  of  attributes  used  by  previous  researchers  and  practitioners.  The  union  of  all  sets 
of  attributes  was  taken  to  form  an  initial  set.  Attributes  were  then  clustered  in  terms  of 
common  orientation  and  purpose.  Redundant  attributes  within  clusters  were  then 
pruned  and  a  consistent  set  of  definitions  chosen. 
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Method 


Five  subjects  participated  in  this  experiment.  Three  worked  as  individuals  and 
two  worked  as  a  team.  The  team  included  an  adaptive  aiding  system  designer  and  a 
former  U.S.  Air  Force  pilot.  The  reason  for  the  team  was  to  enable  participation  in  this 
experiment  of  an  individual  with  substantial  operational  experience. 

The  four  adaptive  aiding  analysts  were  all  very  familiar  with  the  concept  of 
adaptive  aiding  and  the  design  framework  discussed  earlier.  Experience  with  adaptive 
aiding  ranged  from  1  to  15  years,  with  an  average  of  7  years. 

Procedure 


Each  subject,  or  team,  performed  independently  in  separate  rooms.  The 
experiment  was  completed  in  one  day,  averaging  5.5  hours  per  subject  or  team. 


There  were  three  segments  to  the  experiment,  run  in  serial  order: 

1 .  Familiarization  with  Context  -  In  the  first  segment,  subjects  were  asked 
to  read  the  textual,  narrative  mission  scenario.  Subjects  were 
requested  to  take  note  of  significant  mission  events,  since  the  mission 
decomposition  was  not  provided  in  the  familiarization  run.  Note  taking 
was  encouraged  to  facilitate  understanding  of  the  event  sequences  in 
the  text. 

2.  Specification  of  Adaptive  Aiding  -  Once  subjects  had  read  the  scenario 
and  understood  the  context,  the  specification  process  was  begun. 
Subjects  were  given  a  segmented  copy  of  the  scenario  just  read.  Each 
of  the  42  segments  were  similar  in  format  to  Figure  1 .  Subjects  were 
asked  to  specify  adaptive  aiding  using  specification  sheets  that  followed 
the  format  in  Figure  3. 

3.  Rating  Decision  Making  Attributes  -  Using  the  decis:on  making 
attributes  listed  in  Figure  4,  subjects  rated  the  importance  of  these 
attributes  to  the  types  of  aiding  specified  for  each  scenario  event. 


In  summary,  subjects  familiarized  themselves  with  the  context  at  the  outset,  and 
then  produced  42  sets  of  specifications  and  ratings,  one  set  per  scenario  event. 
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RESULTS 

Basic  Summary  Statistics 

Subjects'  specifications  over  all  42  scenario  events  were  compiled  and  summary 
statistics  calculated.  The  summary  specification  statistics  are  depicted  in  Figure  5.  This 
segment  of  the  analysis  focuses  on  the  most  frequent  responses  by  each  subject.  The 
response  categories  of  primary  interest  were  (abbreviation  in  parentheses): 

•  Motivation  for  aiding  (Motive), 

•  Tasks  to  be  aided  (Tasks), 

•  Type  of  aiding  (Type), 

•  Method  of  aid  invocation  (Invocation),  and 

•  Communication  requirements  (Communication). 

Results  showed  that  two  of  the  subjects  (1  and  2)  based  their  adaptive  aiding 
specifications  primarily  on  operator-related  factors  (i.e.,  workload  increase  as  a  result  of 
task  demands,  performance  degradation  due  to  increased  task  demands,  and  explicit 
user  request  for  aiding),  while  subjects  3  and  4  considered  primarily  task-related  factors 
(i.e.,  implementation  practicality  of  aiding,  tactical  significance  of  aiding,  allocation  of 
task  execution  based  on  the  nature  of  the  tasks  to  be  conducted).  These  results 
suggest  that  subjects  1  and  2  were  "human  activity  centered,"  while  subjects  3  and  4 
were  "task  requirements  centered."  In  other  words,  the  former  were  more  concerned 
with  the  operator's  requirements  necessary  for  satisfying  the  task  objectives,  while  the 
latter  were  more  apt  to  consider  the  nature  of  the  task  to  be  completed  according  to 
mission  requirements. 
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Figure  5.  Summary  of  Subjects'  Responses 
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It  is  interesting  to  note  that  the  dichotomy  of  human  activity  centered  vs.  task 
requirements  centered  does  not  hold  if  the  type  of  aiding  chosen  is  considered.  As 
shown  in  Figure  5,  subjects  1  and  2  bracket  subjects  3  and  4  in  terms  of  type  of  aiding 
chosen,  e.g.,  subject  1  chose  “none"  the  least  (5%)  while  subject  2  chose  “none"  the 
most  (43%).  Thus,  the  dichotomy  relates  more  to  why  aiding  is  specified  rather  than 
what  aiding  is  specified.  This  difference  is  further  discussed  in  later  consideration  of 
variations  in  designers'  belief  structures  or  aiding  philosophies. 

The  communication  column  in  Figure  5  also  illustrates  interesting  contrasts.  The 
average  ratings  for  all  subjects  were  high  for  product  information,  i.e.,  what  the  aid's 
outputs  are.  All  but  one  subject  (no.  3)  gave  low  average  ratings  to  process  information, 
i.e.,  how  the  aid  functions.  For  procedural  information,  i.e.,  what  the  aid  is  doing, 
subjects  1  and  2  both  gave  moderate  average  ratings,  while  subjects  3  and  4  were  at 
opposite  extremes.  Thus,  in  the  communication  category,  subjects  1  and  2  were,  again, 
very  similar  in  all  three  average  ratings.  However,  subjects  3  and  4  were  only  similar  for 
one  of  three  average  ratings. 

Discriminant  Analyses 

Discriminant  analyses  were  performed  to  determine  the  extent  to  which  the  type 
of  adaptive  aiding  specified  was  related  to  responses  in  the  other  specification 
categories  in  Figure  3.  This  approach  was  taken  because  subjects'  choices  were  from 
categories  rather  than  continuous  response  variables. 

A  discriminant  model  was  constructed  for  each  subject,  or  team,  using  the  four 
response  categories  for  type  of  aiding  as  the  dependent  variable.  There  were  six 
independent  variables,  including  the  responses  to  the  motive,  tasks,  and  invocation 
categories  and  the  ratings  of  requirements  for  procedural,  product,  and  process 
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information.  Canonical  coefficients  for  the  resulting  discriminant  functions  were 
computed,  which  enabled  ranking  coefficients,  in  terms  of  absolute  values,  to  determine 
relative  influence. 

The  results  are  shown  in  Figure  6.  As  indicated  by  the  boxed  coefficients  in  this 
figure,  subjects  1  and  2  are  very  similar  in  terms  of  the  factors  that  are  primarily 
associated  with  their  specification  decisions.  Subjects  3  and  4  also  have  a  high  degree 
of  similarity.  These  results  are  consistent  with  the  notions  that  subjects  1  and  2  were 
human  activity  centered,  while  subjects  3  and  4  were  tack  requirements  centered.  More 
specifically,  subjects  1  and  2  were  similar  (as  were  subjects  3  and  4)  in  terms  of  the 
variables  they  took  into  account  to  make  decisions.  However,  as  noted  in  the 
discussion  of  Figure  5,  these  pairs  of  subjects  did  not  necessarily  reach  similar 
decisions  for  type  of  aiding. 

Figure  7  indicates  the  goodness  of  fit  of  the  discriminant  models.  Percentage 
agreement  of  predicted  choices  of  types  of  aiding  and  actual  choices  was  60%,  81%, 
74%  and  83%  for  subjects  1  -4,  respectively.  The  average  was  75%,  which  for  this  type 
of  study  is  generally  viewed  to  be  a  good  fit. 

Clearly,  the  discriminant  models  match  the  allocation  decisions  better  than  those 
for  partitioning  and  transformation.  Similarly,  the  models  match  partitioning  decisions 
better  than  those  for  transformation.  These  differences  are  probably  due  to  allocation 
being  a  rather  crisp  decision  compared  to  partitioning  and  transformation.  For  example, 
transformation  can  include  many  concepts  for  modifying  a  task  while  allocation  includes 
just  one  concept  --  automation. 
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Oecislon  Making  Attributes 

Subjects’  ratings  of  decision  making  attributes  were  analyzed  in  the  following 
way.  A  mean  rating  for  each  attribute  was  obtained  across  events  for  each  subject.  To 
assure  that  attribute  ratings  were  not  correlated,  the  set  of  ratings  for  each  subject  were 
analyzed  via  Mann-Whitney  pairwise  comparisons.  Ail  pairwise  comparisons  for 
independence  proved  significant,  rejecting  the  null  hypothesis  that  the  ratings  were  from 
the  same  population.  The  mean  ratings  were  then  normalized  to  facilitate  comparison 
across  subjects.  The  normalized  ratings  were  rank  ordered  to  determine  the  most 
influential  attributes  across  specifications. 

The  top  6  attributes  of  each  subject  were  selected  for  comparison.  After  the  sixth 
attribute,  ratings  tended  to  vary  more  widely.  The  resulting  highly  ranked  attributes  are 
shown  in  Figure  8.  The  rankings  across  subjects,  the  right  column,  were  compiled  by 
ordering  weighted  sums  of  individual  subject's  rankings.  Due  to  the  limited  size  of  the 
subject  population,  no  attempt  was  made  to  statistically  compare  rank  orderings. 


Figure  8.  Decision  Attribute  Rankings  by  Subject 

Comparing  the  rankings  of  subjects  1  and  2  with  those  of  subjects  3  and  4, 
attributes  addressing  the  anticipated  reliability  of  the  aiding  and  availability  of  technology 
to  support  aiding  were  the  top  two  concerns  for  both  groups.  However,  the  two  subject 
groups  differed  on  the  attribute  rankings  after  the  top  two  attributes.  From  this 
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perspective,  the  attribute  rankings  suggest  that  there  was  not  as  distinct  a  grouping 
effect  as  was  earlier  indicated. 

For  example,  attributes  addressing  the  ease  of  introduction  of  aiding  (i.e.,  how 
aiding  is  invoked),  possible  difficulty  of  implementing  the  aiding,  operator-aid 
communication  issues,  and  anticipated  user  attitude  towards  aiding  (i.e.,  attributes  7,  3, 
5,  and  8)  completed  the  ordering  for  the  first  group. 

The  remaining  attributes  for  the  second  group  did  not  reflect  the  first  group’s.  For 
the  second  group,  decision  attributes  pertaining  to  user  attitude,  information 
requirements,  user  interaction  criterion,  and  ease  of  introduction  of  aiding  (i.e.,  attributes 
8, 9, 1 ,  and  7)  completed  the  second  group's  ordering. 

Thus,  the  two  groups  were  similar  except  subjects  1  and  2  emphasized 
implementation  difficulty  and  operator-aid  communication  while  subjects  3  and  4 
focused  on  information  requirements  and  desired  aiding  intervention  criterion.  Clearly, 
the  two  groups  are  not  as  discriminate  as  they  were  in  earlier  analyses.  This  is  likely 
due  to  the  fact  that  the  types  of  aiding  chosen,  and  hence  the  attributes  of  most 
importance,  did  not  follow  this  dichotomy  of  groups. 

Given  the  number  of  subjects  involved  in  the  study,  it  is  difficult  to  conclude  more 
than  the  distinction  mentioned  in  the  previous  paragraph.  This  segment  of  the  study 
shows  that  the  concerns  of  the  designer  corresponds  closely  with  the  type  of  aiding 
desired.  A  few  research  questions  arise  out  of  this  initial  decision  analysis.  Specifically, 
were  the  proper  decision  attributes  posed  to  the  designers?  Possibly  the  attributes 
associated  with  this  process  are  tacit  knowledge  for  designers,  and  would  therefore  be 
difficult  to  characterize.  Further,  are  there  distinctions  between  attributes  valued  based 
not  only  on  type  of  aiding  chosen,  but  also  on  the  requisite  knowledge  brought  to  the 
design  process  by  individual  designers?  A  summarization  of  all  results  (including  the 
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discriminant  analysis  and  the  decision  attribute  analysis)  and  a  possible  interpretation  of 
these  results  are  posed  in  the  Discussion  section  of  this  paper. 

Design  Rule  Elicitation 

In  order  to  gain  further  insight  into  subjects'  decision  making,  each  subject  was 
debriefed  upon  completion  of  the  experiment.  During  this  debriefing,  subjects  were 
queried  about  possible  design  rules  that  may  have  surfaced  in  the  course  of  the 
specification  experiment.  At  least  four  "If-Then”  design  rules  were  elicited  from  each 
subject,  some  with  interesting  implications  for  generating  aiding  specifications.  The 
topics  addressed  ranged  from  the  use  of  specific  aiding  types  under  certain  conditions 
to  how  the  nature  of  operator-aid  communication  varies  with  the  mission  timeline.  The 
complete  set  of  design  rules  can  be  found  in  Appendix  C. 

Most  of  the  rules  were  of  a  general  nature  (e.g.,  IF  pre-occupying  events  occur, 
THEN  aid  background  tasks  according  to  change  in  performance).  Additionally,  most  of 
the  rules  (e.g.,  IF  pre-occupying  events  occur,  THEN  aid  background  tasks  accord'  g  to 
changes  in  performance)  appeared  to  apply  to  the  designer's  particular  approach  to 
aiding  design,  not  to  a  particular  design  philosophy.  Or  stated  more  clearly,  the  rules 
may  reflect  fundamental  principles  of  aid-operator  interaction,  and  not  context 
dependent  implementation  rules. 

The  rules  not  only  provide  insight  into  a  subject's  design  orientation  and 
specification  strategy,  but  may  also  provide  a  basis  for  eventual  development  of  a 
specification  knowledge  base  to  assist  designers  in  specifying  adaptive  aiding  systems. 
These  rules  were  also  used  in  a  post-hoc  analysis  of  belief  systems  possibly  used  by 
subjects  during  the  experiment.  The  belief  systems  are  discussed  in  the  following 
section. 
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DISCUSSION 

The  data  indicate  that  subjects  were  highly  consistent  in  their  specification 
decisions.  This  was  particularly  for  allocation  and  partitioning,  but  less  so  for 
transformation.  There  were  also  substantial  differences  among  individuals,  although 
this  was  not  as  pronounced  in  the  analysis  of  decision  making  attributes. 

Designers'  information  needs  for  making  specification  decisions  are 
demonstrated  by  the  results  in  Figures  6  and  8  and  associated  discussions.  Designers 
are  clearly  interested  in  information  about: 

•  Relationships  among  tasks  and  appropriate  types  of  aiding  (Fig.  6), 

•  Appropriate  invocation  criteria  for  different  types  of  aiding  (Fig.  6), 

•  Appropriate  motivations  for  different  types  of  aiding  (Fig.  6), 

•  Anticipated  reliability  of  aid  behavior  in  normal  vs.  novel  situations  (Fig. 

8),  and 

•  Availability  of  technology  to  support  aiding  implementation  (Fig.  8). 

Designers  appear  to  be  much  less  interested  in  information  about: 

•  Tradeoffs  between  costs  of  communicating  vs.  aiding, 

•  Necessary  level  of  aid  tailorability,  and 

•  Number  and  applicability  of  interface/aiding  models  available. 

These  conclusions  would  appear  to  have  important  implications  for  the  types  of 
research  studies  whose  results  designers  would  value.  In  particular,  from  Figure  6  it 
can  be  concluded  that  designers  are  likely  to  value  data  that  compare  types  of  aiding 
and  appropriate  invocation  criteria  as  a  function  of  types  of  tasks  and  the  motivation  for 
aiding  (e.g.,  likely  performance  decrements  vs.  possibly  excessive  workload).  Further, 
based  on  Figure  8  we  can  conclude  that  they  are  concerned  that  approaches  to  aiding 
be  sufficiently  robust  to  be  supportive  in  a  range  of  situations  and  that  supporting 
technology  be  tested  and  practical.  From  this  perspective,  designers  are  not  likely  to 
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value  research  results  that  simply  show  that  performance  is  better  with  aiding  than 
without  it  --  they  would  like  to  know  the  specific  ranges  of  conditions  where  a  particular 
type  of  aiding  is  valuable. 

These  conclusions  have  important  implications  for  designing  adaptive  aiding 
systems  and  supporting  the  design  process.  The  summary  statistics,  results  of  the 
discriminant  analyses,  and  the  rank  orderings  of  attribute  ratings  show  what  information 
designers  choose  to  use  in  specifying  aiding.  These  results  also  show  what  information 
they  do  not  use.  Clearly,  a  design  support  environment  should  provide  what  is  needed 
and  wanted,  and  not  burden  the  design  process  with  additional  information. 

It  is  also  apparent  that  designers  want  specific,  concrete  information  that  enables 
decision  making.  General  principles  are  only  useful  to  the  extent  that  they  can  be 
readily  translated  into  context-specific  decisions.  Thus,  for  example,  "look  before  you 
leap"  is  an  acceptable  general  principle,  but  "look  for  a  50%  increase  in  response 
latency  before  you  automatically  invoke  aiding"  is  a  more  useful  design  guideline. 

As  a  means  of  integrating  all  of  the  results  presented  in  this  paper,  the 
interpretations  compiled  in  Figure  9  are  offered.  These  interpretations  represent  a 
qualitative  integration  of  all  of  the  statistical  results  presented  earlier,  as  well  as 
designers'  rules  discussed  in  detail  in  Appendix  C.  We  speculate  that  these  differences 
in  beliefs  underlie  the  individual  differences  identified  in  the  results.  Further,  the  fact 
that  subjects  do  not  neatly  fit  in  just  one  row  (i.e.,  one  belief  type)  of  Figure  9  may 
explain  why  differences  among  groups  did  not  consistently  emerge.  For  example,  the 
agreement  of  subjects  1  and  2  on  why  aiding  is  needed,  but  their  disagreement  on  what 
aiding  is  needed  may,  at  least  in  part,  be  explained  by  the  interpretations  in  Figure  9. 
However,  at  this  point,  we  offer  only  the  speculation  that  designers'  beliefs  or  aiding 
philosophy  (explicit  or  otherwise)  is  likely  to  affect  their  design  choices  (i.e., 


31 


Contract  No.  F33615-88-C-3612 
Report  No.  NAWCADWAR-92086-60 


specifications)  as  well  as  the  information  that  they  choose  to  employ  in  making  these 
choices. 
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Belief 

Type 

Type  of  Aiding 

Allocation 

(A) 

Partitioning 

(P) 

Transformation 

(T) 

1 

Let  aid 
execute  well- 
defined  task. 

(SI,  S3,  S4) 

Aid  what  tasks 
pilot  cannot 
attend  to  in 
complex  task. 
(S2,  S4) 

Transform 
difficult  manual 
task. 

(S3.S4) 

II 

Only  allocate 
task  when 
operator 
cannot 
attend  to  it. 

<S2) 

Leave  ill-defined 
parts  of  complex 
task  to  human, 
aid  all  else. 

(SI,  S3) 

Transform 
difficult  situation 
assessment  or 
planning  task. 
(S1.S2) 

Note:  Parentheses  indicate  subjects. 

Figure  9.  Alternative  Belief  Structures  Influencing  Specification  Strategies 


This  speculation  quite  naturally  raises  another  research  issue  --  what  belief 
system  is  appropriate?  More  specifically,  how  should  designers  think  about  aiding 
decisions?  While  the  "correct"  answer  to  this  question  is  not  clear,  it  is  clear  that  the 
answer  is  likely  to  affect  the  types  of  information  sought  and,  consequently,  the  types  of 
aiding  chosen. 


CONCLUSION 


An  investigation  into  designers'  decision  making  processes  involved  in  specifying 
adaptive  aiding  systems  was  conducted.  A  small  population  of  aiding  system  designers 
was  asked  to  analyze  a  scenario  involving  an  advanced  tactical  aircraft  concept  for 
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possible  adaptive  aiding  application.  Two  analyses  of  the  resulting  data  were 
conducted  which  involved  identification  of  most  frequent  responses  for  specification 
categories  and  also  a  ranking  of  the  most  highly  influential  decision  attributes  used 
during  the  specification  process. 

Results  indicated  a  high  degree  of  specification  consistency  for  individual 
designers,  but  a  great  deal  of  variation  among  designers.  Upon  further  analysis,  it 
appeared  that  consideration  of  the  individual's  design  philosophy  (human-activity  or 
task-requirements  centered)  provided  a  unifying  structure  for  interpreting  ail  of  the  data. 

Although  this  was  a  limited  study  conducted  with  a  small  sample,  the  results  are 
encouraging.  Implications  for  further  studies  of  designer  behavior  and  also  integration 
of  later  results  into  automated  tools  for  design  assistance  show  promise. 
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A.1  Adaptive  Aiding  Flight  Scenario  Generation 
A.1.1  Purpose 

Scenario  generation  is  an  integral  part  of  the  crew  systems  design 
process.  During  the  initial  requirements  and  specification  stages,  the  flight 
scenario  serves  as  both  a  vehicle  for  understanding  the  operational  capabilities 
of  the  new  aircraft  and  an  initial  indicator  of  systems  functionality  for  systems 
designers  to  use  in  formulating  detailed  design  requirements. 

Scenarios  are  generated  as  part  of  the  initial  requirements  analysis 
process.  An  understanding  of  the  system  requirements  is  gained  by  the  review 
of  documents  provided  by  the  sponsoring  agency  (usually  the  government 
branch  responsible  for  managing  the  overall  contract).  This  leads  to  the 
generation  of  scenarios  which  reflect  system  requirements,  derivation  of 
functionality,  information  needs,  and  control-display  requirements  for  the  aircraft. 

Two  scenarios  were  generated  by  Midwest  Systems  Research  for  use  on 
this  contract.  However,  only  one  (i.e.,  the  beyond-v  sual-range  (BVR)  scenario) 
was  used  as  experimental  material  during  this  study.  In  the  following  sections, 
the  background  material  used  in  constructing  the  scenario  events  and  method 
used  to  generate  the  events  are  described.  The  task  analysis  method  used  in 
employing  the  method  is  discussed,  as  well  as  the  enhancements  that  were 
added  to  the  events  to  property  represent  the  task  environment  for  aiding.  The 
complete  event  listing  for  the  beyond-visual-range  scenario  is  provided  at  the  end 
of  this  appendix. 

A.1 .2  Source  and  Method 

The  scenarios  were  generated  based  on  the  operational  expertise  of 
Midwest  Systems  Research's  pilot  factors  engineers.  Two  scenarios  were 
generated:  an  advanced  fighter  strike  mission  and  an  beyond  visual  range  air-to- 
air  attack  mission. 

The  scenario  selected  for  the  strike  mission  paralleled  portions  of  those 
developed  in  support  for  Low  Altitude  Navigation  and  Targeting  Infrared  for  Night 
(LANTIRN)  and  Low  Altitude  Night  Attack  (LANA)  systems  integration  and 
evaluation  work.  These  systems,  in  the  F-16C/F-15E  and  A-7,  respectively, 
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involved  the  specifics  of  systems  integration  and  operation  in  full  and  part-task 
simulation  and  later,  flight  test.  This  detailed  work  and  the  more  generic  GFE 
scenarios  were  enough  alike  in  terms  of  requirements  to  make  them  compatible. 

The  air-to-air  mission  as  described  in  the  GFE  documents  was  similar  to 
those  being  worked  in  the  Integrated  Control  and  Avionics  for  Air  Superiority 
(ICAAS)  program  with  which  Midwest  Systems  Research  has  been  associated. 
The  BVR  aspects,  dealing  with  target  acquisition,  prioritization,  and 
reprioritization  present  a  challenge  to  the  systems  design  community  and  aircrew 
members  alike.  Accordingly,  these  tasks  were  selected  as  likely  candidates  in 
need  of  adaptive  aiding  technology. 

A.2  Scenario  Task  Analysis  for  Adaptive  Aiding 

A.2.1  Introduction 

In  order  to  properly  specify  adaptive  aiding  within  the  scenario  context, 
decomposition  of  the  scenario  into  manageable  pieces  was  necessary.  The 
scenario,  as  delivered,  was  written  in  the  third  person  and  consisted  of  a  text 
narrative  describing  the  specific  year  2000+  mission. 

First,  the  scenario  was  segmented  according  to  decomposable  event 
occurrences  in  the  text.  A  decomposable  event  was  defined  as  a  significant 
change  in  pilot  focus,  noticeable  environmental  change  requiring  pilot  input,  etc. 
Next,  the  scenarios  were  task  analyzed  using  the  Identification  of  Advanced 
Technology  Crew  Station  Decision  Points  and  Information  Requirements  report 
breakdown  (Cohen,  1990).  This  document  allowed  for  a  systematic  analysis  of 
the  events  via  mission  phase  and  concurrent  task  representations. 

After  the  first  iteration  on  the  scenarios,  it  became  evident  that  the 
breakdown  was  not  specific  enough  for  adaptive  aiding  analysis.  It  appeared  that 
some  important  information,  both  task  and  operator  related,  was  missing.  Recall 
that  the  purpose  of  this  analysis  was  to  provide  a  well  structured  breakdown  of 
the  scenario  from  wlvch  designers  could  specify  opportunities  for  aiding. 
Towards  this  end,  three  deficiencies  were  identified: 

No  indication  of  attentional  focus  -  Under  the  resulting  task  analysis,  there 
were  no  provisions  for  the  analyst  to  indicate  where  the  pilot’s  attention  was 
currently  focused.  In  particular,  there  were  no  provisions  for  documenting  when 
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a  novel  event  would  (in  the  analyst's  opinion)  capture  the  pitot's  attention  (e.g., 
when  a  high  priority  situation  assessment  task  would  demand  pilot  input,  steering 
attention  away  from  the  currently  active  tasks,  etc.).  Since  aiding  can  be  of 
significant  value  in  this  type  of  situation,  a  notation  was  necessary  to  highlight  the 
difference  in  attention  levels  related  to  specific,  concurrent  tasks. 

Missing  representation  of  the  novel  task  -  Scenarios  were  decomposed 
into  specific  "events"  during  the  analysis.  Events  were  marked  by  a  novel  task 
occurrence  in  the  environment.  There  was  no  notation  available  to  the  analyst 
allowing  for  a  new  task  (or  task  type)  representation.  The  task  analysis 
breakdown  was  available.  However,  a  more  general  notation  was  necessary  to 
show  task  type. 

Confusion  over  which  "actor*  was  executing  the  task  -  In  the  scenario,  a 
certain  level  of  automation  is  already  assumed  (recall  that  the  year  is  "2000+").  A 
more  careful  delineation  of  who  was  executing  a  task  (either  human  or  computer) 
was  necessary.  There  were  two  approaches  available:  either  delineate  directly 
between  human  and  computer  or  ensure  that  the  general  task  type 
unambiguously  indicated  the  actor  in  each  event. 


A.2.2  Modified  Analysis  Method 

A  modified  task  analysis  method  was  developed  to  compensate  for  the 
deficiencies.  The  idea  was  to  produce  accurate  task  breakdowns  for  aiding 
analysis  without  significantly  increasing  representation  complexity.  The  approach 
developed  consisted  of  using  the  task  representations  of  Cohen  (1990)  and 
enhancing  it  with  Rouse's  general  user-system  task  taxonomy  (Rouse,  1991). 
Further,  the  task  listings  were  broken  out  into  foreground  vs.  background  tasks 
allowing  the  analyst  to  show  where  the  pilot's  attention  was  currently  focused. 
Finally,  events  that  consisted  of  both  automated  and  manual  tasks  were 
segmented  and  isolated  so  that  the  actor  in  the  event  was  dear.  Each  of  the 
modifications  is  discussed  below. 

General  User-System  Task  Taxonomy  -  Rouse  (1991)  describes  three  general 
user-system  tasks  present  in  any  human-machine  system  (Figure  1).  Execution 
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and  monitoring  consists  of  carrying  out  an  accepted  plan  of  action,  monitoring  the 
execution  of  the  plan,  and  evaluation  of  the  plan's  success.  Situation 
assessment :  information  seeking  consists  of  generation  and/or  identification  of 
information  sources,  and  evaluation  and  selection  of  a  source  to  satisfy  the 
seeker's  goal.  Situation  assessment:  explanation  involves  these  activities  as 
applied  to  the  alternative  explanations  available  for  phenomena.  Finally, 
planning  and  commitment  focuses  on  the  possible  courses  of  action  and 
possible  operator  tasks  on  them. 

This  taxonomy  characterizes  the  general  tasks  being  undertaken  in  any 
scenario  event.  It  allows  the  analyst  to  quickly  evaluate  what  type  of  task  is 
being  undertaken  and  fit  the  appropriate  type  of  aiding  to  the  event. 

Foreground-Background  Task  Breakdown  -  Task  characterizations  were  further 
categorized  as  either  foreground  or  background  tasks.  A  foreground  task  was 
defined  as  that  novel  task  in  the  event.  The  pilot's  attention  was  oriented  towards 
the  task  as  a  results  of  its  occurrence  in  the  scenario.  Background  tasks,  on  the 
other  hand,  are  those  tasks  that  are  still  critical  in  the  environment,  but  are 
currently  not  the  focus  of  attention  (e.g.,  flight  tasks  are  relegated  to  background 
status  during  a  complex  situation  assessment  task,  etc.).  In  the  resulting 
representation,  foreground  tasks  were  denoted  by  a  single  graphic  box  around 
the  task  listing,  background  tasks  were  denoted  by  a  double  graphic  box  around 
the  task  listing.  No  attempt  to  estimate  how  long  attention  would  remain  shifted 
was  made. 

Event  Characterization  By  Actor  -  There  were  several  methods  available  for 
identifying  the  actor  in  each  event.  We  chose  to  decompose  the  events  further 
based  on  the  actor  instead  of  introducing  a  new  representation  for  event  actor. 
This  approach  allowed  for  finer  grained  event  representation  and  consistent  actor 
identification  without  another  piece  of  notation  in  the  analysis. 

This  enhanced  method  appears  to  have  increased  the  amount  of 
information  necessary  to  specify  adaptive  aiding  for  the  scenario  events  without 
increasing  representation  complexity.  It  also  addresses  the  identified 
deficiencies  in  the  base  task  analysis  representation.  It  facilitates  understanding 
of  the  event  in  terms  of  the  adaptive  aiding  analyst’s  perspective  and  will  also 
support  automated  assistance  in  analyzing  future  scenarios.  The  Beyond  Visual 
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Range  Attack  Scenario  was  analyzed  using  this  method  and  used  during  the 
specification  experiment.  The  complete  scenario  event  breakdown  is  presented 
in  the  following  section. 
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A.3  BEYOND-VISUAL-RANGE  SCENARIO  EVENTS 

This  section  contains  the  complete  event  description  used  during  this 
study.  All  section  headings  and  material  are  unchanged  from  the  original 
stimuli  used. 


INTRODUCTION 

This  brief  Beyond  Visual  Range  (BVR)  scenario  has  been  prepared  to  address 
some  of  the  issues  that  will  arise  in  the  BVR  environment  and  differ  significantly 
from  what  we  have  experienced  in  the  past. 

Recent  improvements  in  long  range  detection  and  identification  techniques, 
combined  with  longer  range  tracking  weapons  and  improved  fire  control 
systems,  may  make  BVR  combat  feasible  in  future  conflicts.  Similar 
advancements  by  potential  adversaries  may  make  it  a  necessity. 

BACKGROUND 

From  the  beginning,  virtually  all  air  combat  has  been  conducted  in  the  "within 
visual  range"  arena,  due  in  part  to  weapons  limitations,  but  to  a  greater  extent 
by  the  requirement  for  a  visual  ID. 

In  the  1950's-1 960's  time  frame  a  type  of  BVR  combat  was  envisioned  in  Air 
Defense  operations.  In  this  scenario,  international  tensions,  point  of  origin, 
route  of  flight,  force  size  and  formation,  and  other  dues  were  considered 
adequate  to  determine  intent.  Air  Defense  Command  units  close  to  our  borders 
were  prepared  to  launch  intercept,  and,  if  necessary,  fire  on  intruders  in  night 
and  all  weather  conditions.  As  it  turned  out,  happily,  all  of  these  things  never 
occurred  at  the  same  time  and  the  result  of  most  ADC  launches  from  the  alert 
hangers  was  to  identify  and  monitor  the  activities  of  possibly  hostile  aircraft.  If 
an  attack  had  been  made,  however,  ADC  was  prepared  to  conduct  BVR 
operations  using  a  combination  of  ground  based  radar  stations  (netted)  and 
aircraft  carrying  airborne  attack  radars  to  direct  air-to-air  fighter  aircraft  to  within 
their  own  radar  range  and  guided  and  unguided  missiles. 

Between  Korea  and  Vietnam  there  grew  the  feeling  that,  with  tracking  radar  and 
guided  missiles,  air  combat  would  be  done  at  greater  ranges  and  that  the  tail 
chase  and  shoot  ’em  were  things  of  the  past.  Some  of  these  feelings  were 
shelved  fairly  early  in  the  Vietnam  experience  when  then  Col.  Fred  Btesse, 
commander  of  the  F-4  wing  at  DaNang  sent  a  note  to  the  Pentagon  -  "I  need  a 
gun." 

In  the  following  paragraphs,  we  will  describe  a  segment  of  a  generic  BVR 
encounter  focusing  on  some  of  the  things  that  will  be  unique  to  this  kind  of  air 
combat  in  terms  of  pilot  decisions  and  activities  from  initial 
detection/identification  through  the  approach  to  merge  where  weapons  and 
tactics  will  revert,  essentially,  to  those  experienced  today.  Topics  to  be 
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addressed  include  many  of  those  covered  in  the  NADC  draft  document, 
Identification  of  Advanced  Technology  Crew.  Station  Decision  Points  and 
Information  Requirements.  January  1990.  Special  emphasis  has  been  placed 
on  target  search/identification,  target  assignment,  reassignment  and  attack. 
These  functions  appear  in  the  document  under  Combat  Air  Patrol  from  5.0 
Assume  Cap  (pg.  27)  5.1.4  -  5.1.6,  5.4  Preliminary  Raid  Assessment  (pg.  28) 
5.4.1  -  5.4.3, 6.  Intercept  (pg.  29)  6.1.3  -  6.1.7, 6.4,  Raid  Assessment  (Sorting) 
(pg.  30)  6.4.1  -  6.4.7,  to  7.0  Attack  (pg.  31)  7.1.3  -  7.1.5  and  7.1.8.  Similar 
functions  are  called  out  in  the  same  portions  of  the  Deck  Launched  Intercept 
mission. 

Before  a  flight,  pilots  are  briefed  in  detail  on  the  mission,  how  it  should  be  flown 
and  what  to  expect  in  the  line  of  weather,  the  threat  (air  and  ground),  areas  of 
relative  safety,  standoff  support,  location  and  schedule  for  air  refueling  and 
others.  As  long  as  the  mission  proceeds  in  accordance  with  what  is  expected, 
the  challenge  is  relatively  low.  It  is  when  one  or  more  of  the  mission  variables 
changes  significantly  that  things  can  get  extremely  complicated.  Several  key 
technologies  are  being  examined  to  help  crewmembers  in  these  situations. 
Major  areas  of  concern  that  we  have  encountered  in  a  number  of  programs, 
some  of  which  have  involved  fairly  high  fidelity  simulation,  are: 

a.  Multiple  sensor  management/control  and  interpretation. 

b.  Use  and  control  of  increasingly  sophisticated  self-protection 
sensors,  controls,  and  systems. 

c.  Availability  of  a  wider  range  of  weapons  with  varying 
capabilities  and  applications. 

d.  Use  of  increasingly  sophisticated  command  and  control 

systems. 

e.  All  of  the  above  combined  to  expand  the  need  for 
crewmember  awareness  or  concern  with  increasing 
volumes  of  information  and  things  to  control. 

Begin  scenario: 

NPlfll 

The  initial  BVR  analysis  was  conducted  as  follows:  Each  significant 
scenario  event  was  broken  out  and  identified  as  one  of  the  three  general  user- 
system  tasks  (situation  assessment,  planning  and  committment,  or  execution 
and  monitoring)  taken  from  Rouse.  Appropriate  tasks  were  then  extracted  from 
the  "Identification  of  Advanced  Technology  Crew  Station  Decision  Points  and 
Information  Requirements  report  by  David  Cohen  and  attached  to  significant 
scenario  events.  This  listing  ensures  that  the  analysts  later  specifying 
opportunities  for  Adaptive  Aiding  are  working  from  the  same  set  of  operator 
tasks  in  each  event. 
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User-system  task  type  headings  are  abbreviated  as: 

sa:  Situation  Assessment 
Subtasks: 

Information  seeking 
Explanation 

pc:  Planning  and  Committment 
Subtasks: 

troubleshooting 
em:  Execution  and  Monitoring. 

Foreground  tasks  are  enclosed  in  single  line  boxes.  Background  tasks 
are  enclosed  in  double  line  boxes.  New  background  task  boxes  will  only  be 
present  when  there  is  significant  change  in  the  task's  status. 

Combat  Air  Patrol  (CAP) 

In  this  scenario,  Blue  Right  of  four  fighters  has  launched  on  a  mission  to 
proceed  to  a  preplanned  CAP  orbit  under  the  control  of  CROWN,  a  standoff 
Airborne  Command,  Control  and  Communications  (ABCCC)  type  of  platform.  In 
this  case,  the  Command,  Control,  Communication  and  Identification  (C3l) 
environment  could  include  the  ABCCC,  JTIDS,  perhaps  SATCOM  or  a 
combination  of  all  of  them.  In  any  case,  there  is  external  support  that  will  aid  in 
target  and  threat  detection  plus  identification.  This  support  will  also  provide  at 
least  initial  target  assignments  at  the  flight  level  as  the  flight  develops.  All 
communications  between  CROWN  and  Blue  Right  will  be  via  the  secure  net 
except  in  a  bona-fide  emergency. 


Assumed  automated: 

5.0  Assume  CAP 

5.1.2  Select  pilot  relief  mode 
5.1 .7  Activate  mission  recorder  system 
5.1.6  Determine  frequency  of  visual  search 
5.6.4  Set  EMCON 


5.1.1  Control  Aircraft 

5.1.3  Monitor  systems  status 
5.1.5  Set  Formation 

5.5.1  Monitor  position 

5.5.2  Monitor  course 

5.5.3  Monitor  speed 

5.5.4  Monitor  altitude 
S^SJ^erformjTavs^slem^gd^ 


...  3 fll 
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JEnroute  to  the  orbit,  Blue  Flight  receives  targeting  information  from  CROWN. 

I  5.6.5  Perform  SATCOM  ~  1 

2.  sa: 

|A  group  (two  formations  of  four),  of  probably  hostile  aircraft,  are  approaching 
from  the  northeast  at  a  range  of  approximately  200  NM. 


5.3  Coordinated  sensor  activities 

5.3.2  Correlate  on-board  data/information 

5.3.3  Correlate  external  data  with  on-board 
data/information 


3.  sa:  Information  seeking  and  explanation 

{Positive  identification  is  not  available  yet,  but  their  speed  (in  excess  of  Mach  1 ) 
indicates  fighters  possibly  configured  for  air-to-air  combat.  There  is  no 
correlation  with  known  friendly  forces  and  their  present  course  indicates  that 
they  could  have  been  launched  to  disrupt  the  CV  Task  Force  air  defenses  in 
preparation  for  a  direct  attack  on  the  CV  Task  Force  itself. 

5.2  Response  to  threat  ™-~-~ 

5.2.2  Determine  threat  degree 

5.2.3  Determine  Imminence  of  threat 

5.3.4  Interpret  sensor  data/infonmation 

5.4  Preliminary  raid  assessment 

5.4.1  Perform  target  search/detection 

5.4.2  Perform  target  acquisition 

5.4.3  Perform  target  ID/classification 
_ 5.6.5  Perform  SATCOM 


4.  sa;  Information  seeking 

JCROWN  provides  as  much  targeting  information  as  possible  as  the  two  forces 
close  to  about  150  miles.  This  information  is  transmitted  to  the  Blue  Flight's 
aircraft  fire  control  systems  via  secure  link  where  it  appears  on  each  aircraft's 
tactical  situation  display. 


6.0  Intercept 

6.3.3  Correlate  external  data  with  on-board  data/information 

6.4.2  Perform  target  acquisition 

6.4.3  Perform  target  ID 

6.4.4  Assess  raid 

6.4.5  Determine  target  assignments 

6.4.6  Determine  preliminary  targeting _ 
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6.1.5  Maintain  formation/mutual  support 


5.  em; 

| As  this  is  being  done,  the  members  of  Blue  Flight  activate  and  verify  the 
operation  of  their  threat  warning  and  self-protection  systems  and  check  their 
weapons  readiness  status. 


6.1.4  Monitor  weapons  status 

6.2.1  Monitor  threat  detection  systems 


6.  em; 

|So  as  not  to  advertise  their  presence,  Blue  Flight  has  been  operating  in  a  very 
quiet  mode  limiting  emissions  to  only  those  required  for  intra-flight  coordination. 


(this  event  is  obviously  out  of  order) 
5.0  Assume  CAP 
5.6  Communicate 

5.6.4  Set  EMCON 


7.  pc;  troubleshooting 

(Blue  Lead  becomes  a  little  concerned  that  this  mode  of  operations  may  have  to 
be  abandoned  soon.  He  must  validate/coordinate  CROWN’S  inputs  for  the  rare 
case  that  CROWN  cannot  provide  final  verification  of  target  type  and  precise 
position  and  altitude. 


6.2 


6.1.7  Analyze  tactical  situation 
Response  to  threat 
6.2.3  Determine  imminence  of  threat 


6.3.2  Correlate  on-board  data/information 

6.3.3  Correlate  external  data  with  on-board  data/information 


8.  em; 

|While  there  are  intermittent  threat  warnings  during  this  period  of  time,  none  are 
of  a  persistent  nature,  so  Blue  Right  continues  with  its  self-protection  systems  in 
the  semi-automatic  mode.  (Armed  in  an  automatic  mode,  the  system  might 
unleash  its  fury  on  a  chance,  and  only  temporary,  lock-on.  This  reaction  would 
announce  the  presence  of  Blue  Right  to  everyone.) 

1  6.2.1  Monitor  threat  detection  systems 
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|Biue  Lead's  concern  increases  as  no  activity  is  indicated  from  his  IR  sensors 
and  there  are  no  indications  that  his  wingmen  have  detected  any  either. 
Without  specific  altitude  information  from  CROWN,  this  could  indicate  the  the 
dosing  flight  is  at  a  lower  altitude  than  anticipated,  and  could  be  sacrificing 
altitude  for  the  protection  afforded  by  moisture  in  high  drrus  douds.  While  the 
new  IR  equipment  is  much  better  than  that  used  in  the  1990's,  it  still  succumbs 
to  the  basic  laws  of  physics  and  the  bad  guys  know  this. 


I  6.1.7  Analyze  tactical  situation 

i 

1  6.5.5  Adjust  flight  plan,  as  needed 

■ 

As  the  approaching  groups  close  to  within  approximately  100  miles  of  each 
other,  CROWN’S  information  begins  to  improve. 

10.  sa:  Information  seeking 

|Blue  Flight  receives  good  flight  information  on  the  approaching  aircraft. 


5.6.5  Perform  SATCOM 

6.3.3  Correlate  external  data  with  on-board  data/information 

6.4.4  Assess  raid  _ 


11.  em: 

|The  tactical  situation  display  indicates  their  altitude  to  be  at  FL  350, 
approximately  100  NM  in  front  of  him,  crossing  his  flight  path  at  an  angle. 


6.3.4  Interpret  sensor  data/information 

6.4  Raid  assessment  (sorting) 

6.4.1  Perform  target  search/detection 
_ 6.4.2  Perform  target  acquisition _ 


12,  pc; 

Preliminary  target  assignments  are  made;  Blue  Lead  and  Two  will  take  on  the 
lead  element  (one  the  left)  -  Three  and  Four  will  engage  the  (slightly  trailing) 
flight  on  the  right. 


6.4.5  Determine  target  assignments 

6.4.6  Determine  preliminary  targeting 


13.  sa;  explanation 

|The  aircraft  type  is  confirmed  as  enemy 

"  6.2.2  Determine  threat  degree 

_ 6.4.3  Perform  target  identification/dassification 
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14,-PCI 

|-  Cleared  to  Fire) 


7.0  Attack 

*  7.1.1  Control  aircraft  (now  in  foreground) 
7.2  Response  to  threat 

7.2.5  Determine  to  avoid  or  suppress 
7.4  Final  targeting 

7.4.5  Obtain  clearance  to  fire 


is,  gnu 

| As  Blue  Lead  selects  an  expanded  image  mode  on  his  tactical  display, 

1  7.3.1  Operate  sensors  I 


16.  em; 

(signals  arm  up. 

1  6.6.2  Communicate  secure  voice  1 

17,  emi 

land  initiates  first  target  designation, 

|  7.4  3  Comply  with  targeting  assignments  1 

|the  threat  warning  system  signals  search,  lock  on  and  then  track. 

7.2.2  Monitor  threat  detection  systems 

_ 7.2.4  Determine  imminence  of  threat _ 

The  two  hostile  flights  have  gone  fully  active. 

19.  SSI  explanation 

|Ail  evidence  points  to  the  fact  that  the  enemy  is  fully  aware  of  their  presence 
and  intent. 


7.3.4  Interpret  sensor  data/information 


] 


20.  em: 
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JBIue  Lead  quickly  selects  auto  on  his  self-protection  systems, 

I  7.2.6  Perform  threat  response 

u 

2i.»  asm 

Iswitches  his  radar  out  of  stand-by  to  a  multitarget  search  and  track  mode, 

I  7.3.1  Operate  sensors 

Z3 

22.  sa;  explanation 

land  quickly  confirms  correlations  between  CROWN  and  his  on-board  sensor 
data  -  they  agree. 


7.3.3  Correlate  external  data  with  on-board  data/information 

7.3.4  Interpret  sensor  data/information _ 


23i  pc; 

(With  only  seconds  to  a  maximum  range  firing  solution,  the  opposing  force  starts 
a  turn  to  convert  their  angle-off  heading  to  head  on. 


7. 1 .5  Analyze  tactical  situation 
7.1.8  Analyze  disengagement  criteria 


7.7.5  Adjust  flight  plan,  as  needed 


24.  sa; 

|The  hostile  trailing  group  of  four  aircraft  crosses  behind  the  lead  group  and  the 
two  groups  begin  to  separate  slightly.  This  tactic  was  developed  in  the  mid¬ 
nineties  when  it  was  discovered  that  target  assignment  (or  reassignment)  and 
some  prioritization  was  accomplished  manually  by  the  Flight  leaders.  This 
caused  confusion  and  required  valuable  time  to  accomplish,  allowing  the 
hostile  forces  extra  time  to  penetrate  into  our  defenses,  maneuver  into  better 
positions  and,  sometimes,  launch  their  weapons. 

I  7.3.4  Interpret  sensor  data/information 

25.  sa;  explanation 

|As  the  two  hostile  groups  begin  to  separate,  two  more  groups  of  four  images 
appear  on  Blue  Right’s  tactical  displays.  These  two  additional  groups  of  hostile 
targets  appear  to  be  clustered  near  to  the  original  two  groups  but  are  slightly 
faster  and  off  to  the  side. 
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7.1.5  Analyze  tactical  situation 

7,3.4  Interpret  sensor  data/infonmation 


26.  .gnu 

|To  confirm  target  acquisition  Blue  Lead  activates  his  decoy  detection  device  to 
confirm  the  new  target  signatures  (radar,  IR,  Laser,  etc  ). 

I  7,3.1  Operate  sensors  1 

27.  sa;  explanation 

{Eight  of  the  sixteen  hostile  targets  disappear  from  the  screen.  Blue  Lead 
confirms  his  eight  hostile  target  aircraft  are  the  same  as  those  identified  by 
CROWN.  -  They  are. 


7.3.3  Correlate  external  data  with  on-board  data/information 

7.3.4  Interpret  sensor  data/information _ 


28.  pc:  by  computer 

{Blue  Flight's  mission  computers  detect  this  evasive  decoy  tactic  and 
automatically  re-prioritize  the  targets. 

1  Assumed  automated  \ 


TSLsmi 

IBIue  Lead  and  Two  are  assigned  the  port  group  of  aircraft,  while  Blue  Three 
and  Four  are  assigned  the  starboard  group  of  four. 


7.4.1  Determine  dynamic  geometry  maneuvers  required 
7.4.3  Comply  with  targeting  assignments _ 


39.-g.rn: 

The  hostile  group  of  four,  assigned  to  Blue  Leader  and  Two,  are  just  inside  the 
lethal  range  of  their  advanced  long  range  air-to-air  missiles  (ALFLAAM)  and  still 
outside  the  hostile  group's  long  range  missiles. 


7.3.4  Interpret  sensor  data/information 

7.4.4  Select  weaponry _ 


33,jgm; 

|Blue  Leader  launches  two  ALRAAMs,  followed  almost  instantly  by  Blue  Two’s 
launch  of  two. 
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7.5  Weapon  delivery 

7.5.1  Select  weapon/weapon  mode 
_ 7.5.2  Committ  weapons _ 


{Blue  Lead  notes  a  warning  that  one  of  his  missiles  has  malfunctioned. 

1  71 .4  Monitor  weapons  status  1 


33.  sa:  explanation 

JFurther  checks  reveal  that  it  never  left  the  launcher  and  the  Fire  Control 
Computer  has  switched  to  Medium  Range  Missiles  and  associated  firing  modes 
and  displays. 


!  7.1.4  Monitor  weapons  status 

7.1.5  Analyze  tactical  situation 

_ 7.5.1  Select  weapon/weapon  mode _ 

34.  em;  by  FCC 

[The  three  ALRAAMs  are  guided  by  CROWN’S  signals  until  midway  to  the  target 
when  they  take  over  their  own  active  guidance  to  their  assigned  targets.  Each 
of  the  three  missiles  are  assigned  to  three  hostile  targets. 


1  7.5.4  Provide  weapon  steering  data/illumination 

[Blue  Three  and  Blue  Four  have  also  launched  four  ALRAAMs  against  their 
respective  group  of  four  hostile  aircraft. 

1  7.1.3  Maintain  mutual  support,  as  required  t 


}The  situation  Display  in  Blue  Leader's  aircraft  signals  the  "kill"  of  three  of  the 
four  hostile  aircraft  in  his  assigned  port  group  with  no  signs  of  hostile  missiles  in 
flight. 


7.6  Damage  assessment 

7.6.1  Determine  target  damage 


37,  pc; 
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A  backup  missile  must  be  used.  At  this  point,  Blue  Leader  and  number  Two  are 
approaching  the  range  of  their  Advancad  Median  Range  Air-to-Air  Missile 
(AMRAAM)  and 

7.6.2  Assess  reattack  options  1 

38.  sa:  explanation,  then  fig; 

|Blue  Leader  detects  the  sole  survivor  of  the  hostile  group  is  in  a  hard  Starboard 
break. 

|He  is  attempting  to  disengage. 


7.1.5  Analyze  tactical  situation 

7.2.5  Determine  to  avoid  or  suppress 


39.  em; 

|Blue  Leader  allows  for  his  automatic  weapon  selection  to  select  and  arm  one  of 
his  AMRAAMs.  As  the  single  hostile  fighter  sweeps  through  Blue  Leader's 
launch  range  in  his  turn  to  run, 

I  7.2.6  Perform  threat  response  | 


| Blue  Leader  launches  his  AMRAAM.  This  missile  guides  and  tracks  to  the  kill. 


7.6.3  Execute  reattack,  as  required 
7.5.2  Committ  weapons _ 


4t.  sm; 

JThe  Blue  Three  and  Four  missile  launches  against  their  starboard  group  of  four 
hostile  fighters  were  successful  on  their  first  launch. 


6.6.3  Perform  D/L  comm  w/friendlies 
7.6  Damage  assessment 
_ 7.6.1  Determine  target  damage 


42.  pc;  CROWN  provides  plan,  Blue  committs 


ICROWN  confirms  the  "kills"  and  provides  vectors  to  a  tanker  orbit  in  the  vicinity 
of  the  Task  Force.  Blue  Flight’s  remaining  missiles  will  be  held  in  reserve 
pending  the  outcome  of  an  intercept  in  progress,  on  a  flight  of  suspected  attack 
aircraft  approaching  the  CTF  from  the  east. 


Q 


6.6.5  Perform  SATCOM 

6.3.4  Interpret  sensor  data/information 
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6.1 .6  Monitor  systems  status 
6.1.8  Monitor  fuel  status 


end  of  scenario. 

The  requirement  for  high  levels  of  sophistication  (automation)  and  decision¬ 
making  can  vary  greatly  from  mission  to  mission.  At  the  beginning,  there  are 
normally  Rules  of  Engagement  (ROE)  that  arrive  from  on  high,  to  which  units 
and  crewmembers  are  expected  to  adhere.  Tactics  are  developed  around 
those  rules  and  tactics  (and  capabilities)  expected  to  be  employed  by  the 
enemy. 


Note:  Some  of  the  data  correlation  exercises  may  be  background  tasks. 
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B.l  Purpose  for  Decision  Making  Characterization 

Although  the  framework  for  aiding  specification  addressed  the  general 
issues  to  be  considered  during  aiding  specification  (see  pg.  7,  body  of  report),  it 
made  no  provisions  for  considering  the  designer's  decision  making  model  used  in 
during  the  process.  A  decision  model  is  obviously  used  in  specifying  any 
complex  system.  Currently,  the  system  designer  conducts  the  decision  making 
process  in  his/her  head  while  specifying  intended  functionality. 

A  goal  of  this  investigation  was  to  determine  what  types  of  information  are 
valued  by  the  designer.  Through  this  process  we  could  identify  the  decision 
process  and  define  methods  and  tools  for  aiding  the  designer. 

To  understand  the  process  of  design  decision  making,  we  began 
formulation  of  a  decision  making  model  of  designers  engaged  in  an  aiding 
specification  exercise.  Most  of  the  modeling  effort  consisted  of  identifying  the 
decision  attributes  that  would  comprise  the  model.  In  the  following  sections,  the 
process  used,  sources  of  decision  attributes,  and  model  development  are 
discussed.  The  final  set  of  attributes  were  evaluated  during  the  investigation. 

B.2  Sources  of  Decision  Making  Attributes 

While  analyzing  adaptive  aiding  specification  parameters,  it  became 
obvious  that  we  cannot  expect  to  aid  the  design  process  without  understanding 
the  decision  space  in  which  these  specifications  are  made.  In  order  to  bound  the 
space  of  possible  decision  attributes,  only  those  concerned  with  the  three  primary 
aiding  design  decisions  were  initially  considered: 

1)  type  of  aiding  to  be  employed, 

2)  whether  aiding  was  appropriate  for  this  application, 

3)  intended  operator-aid  interaction  characteristics. 

A  large  initial  number  of  attributes  (28),  were  compiled.  This  listing  was 
compiled  from  numerous  adaptive  aiding  references  (Primary  references:  Rouse, 
1988;  Rouse,  Geddes,  and  Curry,  1986;  Revesman  and  Greenstein,  1986;  Noah 
and  Halpin,  1986;  Morris,  Rouse,  and  Ward,  1985b;  Andes,  1990;  Andes,  1987). 
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In  addition  to  those  found  in  the  literature,  a  few  were  generated  to  cover  obvious 
voids  in  the  literature  base.  The  generated  set  of  attributes  was  large  and 
unwieldy;  some  of  the  attributes  were  taken  from  the  specification  framework; 
some  not  well  documented.  Most  of  the  attributes  were  not  easily  parameterized, 
and  further,  some  of  the  attributes  appeared  to  be  confounded  with  others.  The 
complete  listing  of  compiled  attributes  is  given  below. 


1 .  Accuracy  of  result  /  performance  required 

In  most  system  applications,  the  accuracy  of  the  result  of  aiding,  or 
minimal  aid  performance  required  is  of  paramount  importance  in  the  decision  of 
whether  to  specify  adaptive  aiding  or  not. 

2.  Acceptability  of  achievable  aid  performance 

Although  several  of  the  attributes  have  bearing  on  this  attribute  (e.g., 
technology  available,  accuracy  of  result,  etc.),  the  acceptability  of  aid 
performance  (from  the  designer’s  perspective)  affects  the  decision  to  specify 
adaptive  aiding. 

3.  Desired  intervention  criterion 

There  are  several  intervention  criterion  upon  which  aid  intervention  is 
based  (e.g.,  unacceptable  operator  performance,  number  of  concurrent  tasks, 
operator  errors).  The  desired  criterion  must  be  considered  in  the  context  of 
desired  aiding. 

4.  Ease  of  aiding  introduction 

The  designer  must  consider  the  ease  with  which  aiding  can  be  introduced 
into  the  task  environment.  For  example,  will  the  operator  perceive  cognitive 
disunity  (i.e.,  "Is  this  a  new  task,  or  an  alteration  of  the  previous  task?")  when  a 
task  transformation  is  introduced? 

5.  Ease  of  aiding  removal 

It  is  also  important  that  the  negative  cognitive  and  perceptual  effects  of 
removal  of  adaptive  aiding  be  minimized.  In  this  respect,  the  designer  must 
consider  the  costs  of  removal  vs.  the  benefits  of  allowing  the  aid  to  execute  a 
task  to  completion  (if  discrete). 

6.  Technology  available  to  support  aiding 

The  designer  must  consider  what  role  technology  push/pull  will  play  in 
implementing  adaptive  aiding  systems. 

7.  Preconfiourabilitv  of  aiding 

Some  adaptive  aiding  subsystems  may  have  to  be  preconfigured  based 
on  the  particular  mission  context.  Alternatively,  it  may  be  impossible  to 
preconfigure  some  functional  aspects  of  the  aid,  specifically  when  an  error 
situation  arises. 

8.  Task  environment  effects  on  aiding 

Based  on  research,  task  recency  effects  (i.e.,  the  task  just  completed 
affects  operator  performance  on  the  current  task)  must  be  considered.  In  these 
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situations,  the  designer  must  plan  for  a  change  in  the  operator  ROCs  (response 
operating  charactenstics). 

9.  Accountability 

The  designer  must  consider  to  what  level  the  system  will  be  accountable 
for  its  actions  and  to  what  degree  the  adaptive  aiding  system  designer  will  be 
responsible  for  aid  behavior. 

10.  Levels  of  user  discretion  available  within  aiding  system 

The  amount  of  user  discretion  can  be  understood  as  parameterization  of 
functionality,  verbosity  and  type  of  feedback  about  aid  activity,  etc. 

1 1 .  Granularity  of  user  control  over  aiding 

The  amount  that  aiding  parameters  can  be  adjusted,  not  just  distinct  levels 
(i.e.,  continuous  adjustmen  rather  than  the  3  levels  of  adaptive  aiding). 

1 2.  Cost  of  communication  vs.  cost  of  aiding  operator  errors 

The  designer  must  consider  this  tradeoff  in  specifying  the  aid/operator 
communication  interface. 

1 3.  Training  vs.  aiding  decisions 

This  attribute  refers  to  the  utility  tradeoff  of  when  to  train  the  operator  vs. 
when  to  provide  adaptive  aiding  on  these  tasks.  Can  be  economically, 
technologically,  or  psychologically  motivated,  to  name  a  few. 

1 4.  Validity  of  aiding 

How  valid  is  adaptive  aiding  technology  within  this  task  context? 

1 5.  Reliability  of  aiding 

How  reliable  can  we  expect  the  adaptive  aiding  to  be  within  this  task 
context? 

16.  Viability  of  aiding 

Is  adaptive  aiding  viable  within  this  task  context?  Can  another  type  of 
aiding  or  system  redesign  solve  the  problem? 

17.  Desirability  of  aiding 

Is  adaptive  aiding  the  desired  solution  within  this  task  context?  Further, 
how  is  this  attribute  value  changed  based  on  the  perspective  (e.g.,  designer  vs. 
user  of  system). 

1 8.  Predictability  of  aiding 

In  order  to  foster  user  acceptance  of  the  system  and  specify  the  aiding 
system,  the  designer  must  have  some  reasonable  level  of  confidence  in  the 
adaptive  aiding  system's  behavior. 

19.  Anticipated  difficulty  of  aid  implementation 

This  attribute  can  be  used  in  the  go/no  go  aiding  specification  by  the 
designer.  In  addition,  the  attribute  can  be  considered  in  terms  of  the  level  of  aid 
functionality  and  level  of  technology  embedded. 

20.  User  attitude  towards  particular  aiding 

Some  types  of  aiding  are  more  acceptable  to  a  user  population  than 
others.  If  the  designer  is  specifying  a  risky  adaptive  akSng  system  from  the 
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user's  point  of  view  after  considering  this  attribute,  there  probably  is  a  valid, 
defendable  reason. 

21.  Number/aoolicabilitv  of  HCI/aidino  models 

Tools,  task  models,  simulations,  etc.  allow  the  designer  to  gain  insight  into 
the  process  that  he  wishes  to  aid.  In  addition,  embeddable  models  may  facilitate 
better  interaction  and  aid  functionality. 

22.  Psychological  comfort  of  aid  presence 

This  is  related  to  fostering  user  acceptance,  however  this  attribute 
addresses  the  more  subtle  effect  of  adaptive  aiding  presence:  "How  does  the 
aid's  presence  increase  the  operator's  confidence?" 

23.  Information  requirements  for  task  aiding 

The  information  requirements  necessary  in  a  task  environment,  both 
information  requirements  of  the  aid  and  information  requirements  of  the  operator 
will  drive  how  and  what  will  be  aided  in  the  system.  For  example,  if  an 
information  requirement  of  a  specified  aiding  system  is  to  get  a  P300  (i.e., 
evoked  response  potential  from  the  brain)  from  the  operator  as  an  enabling 
condition,  it  is  probably  not  reasonable  to  specify  this  particular  aid. 

24.  Aid  tailorabilitv 

How  much  of  the  aid's  behavior  can  be  tailored  based  on  individual 
differences,  population  differences,  or  not  at  all. 

25.  Risks  of  aiding:  Designer.  Technology,  and  User 

The  general  categories  of  risk  are  of  paramount  importance  to  the 
specification  of  an  adaptive  aiding  system.  Although  these  are  more  general 
categories  of  other  specified  attributes,  the  higher  level  interactions  may  be  more 
important  to  specification. 

26.  Granularity  of  aid/operator  communication 

Related  to  communication  vs.  aiding,  this  attribute  must  be  considered 
from  the  viewpoint  of:  "What  level  of  communication  is  necessary  and  what  is 
possible  given  time  and  operational  constraints?" 

27.  "Efficient  frontier  of  aiding  specification/performance 

This  attribute  comes  from  decision  analysis.  Considered  to  establish  what 
optimizations,  tradeoffs,  etc  are  reasonable  between  the  design  and  the  intended 
user  population. 


28.  Level  of  integration  of  aiding  into  avionics 

Is  the  adaptive  aiding  system  to  be  embedded  in  "to  be  built”  systems  or 
as  an  add-on,  etc?  What  cooperation  between  intelligent  systems  can  be 
expected?  This  attribute  refers  more  to  supporting  structure  for  aiding,  but  is  a 
primary  design  concern. 


This  preliminary  list  appeared  to  provide  a  reasonable  basis  for  identifying 
the  salient  decision  attributes  in  aiding  specification.  However,  it  was 
unreasonable  to  assume  that  we  could  evaluate  such  a  large  list  under 
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experimental  conditions.  Structuring  and  categorization  of  the  attributes  was 
undertaken  in  an  effort  to  organize  and  reduce  the  set  of  attributes  before 
evaluation. 


B.3  Decision  Making  Attribute  Categorization 


Upon  analysis  of  the  generated  list,  four  type  categories  emerged. 
Preliminary  categorizations  were: 


Aid  functional  design 
Aid  performance 
Aid-operator  interaction 
Supporting  technology 


The  set  of  28  attributes  ware  reduced  to  12  as  a  result  of  the  analysis  according 
to  the  method  described  in  the  report  body.  Table  B.1  below  depicts  the  design 
attributes  according  to  category. 


Attribute  Category 

Attribute  Name 

Aid  Functional  Design 

Anticipated  intervention  criterion 
Training  vs.  aiding  decisions 

Difficulty  of  aid  implementation 

Aid  Performance 

Reliability  of  aiding 

Types  of  aiding  communication 

Overall  Risk 

Aid-Operator  Interaction 

information  requirements  for  aiding 
Ease  of  aiding  introduction/removal 
User  attitude  towards  aiding 

Level  of  aid  tailorability 

Supporting  Technology 

Available  operator  models 

Available  hardware 

Table  B.1  -  Decision  Attributes  by  Category 

Complete  definitions  for  the  1 2  attributes  used  in  the  investigation  are  provided  in 
the  report  text.  The  next  section  describes  how  the  decision  attributes  were 
used. 
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B.3.1  Role  in  Adaptive  Aidino  Experiment 

As  stated  earlier,  it  was  a  program  goal  to  gain  greater  understanding  of 
how  designers’  decision  making  processes  influence  the  specification  and  design 
of  the  aiding  system.  Towards  that  end,  rating  data  were  collected  on  the 
attributes  during  the  empirical  investigation.  The  data  consisted  of  rating  the 
attributes  in  terms  of  importance  to  the  overall  success  of  each  aid  specified. 

We  probed  the  decision  process  of  subjects  by  requesting  them  to  rate  the 
importance  of  the  12  purported  design  decision  attributes  during  the  specification 
experiment.  A  rating  sheet  of  the  attributes  was  filled  in  by  the  subject 
immediately  following  each  event  specification.  Further  discussion  of  the  data 
collection  process  is  described  in  the  body  of  the  report. 

It  was  suspected  that  the  attributes  would  be  weighted  differently  for 
different  designers  as  well  as  under  different  task  circumstances.  As  discussed 
in  the  results  section  of  the  report,  this  set  of  attributes  appeared  to  cover 
designers’  decision  attribute  space  adequately  -  at  least  for  the  small  population 
studied. 

Refinement  and  validation  of  the  attributes  should  allow  us  to  assist  the 
designer  in  the  initial  specification  phase.  Further,  decision  models  constructed 
for  both  the  designer  and  intended  user  population  should  highlight  preference 
differences,  resulting  in  a  system  that  is  not  only  designed  well,  but  is  easy  to 
use. 
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APPENDIX  C:  DESIGN  HEURISTICS 
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C.1  Introduction 

During  the  experiment  debriefing,  subjects  were  questioned  about  their 
approach  to  design,  in  particular,  the  experimenter  inquired  about  design  rules 
that  subjects  developed  during  the  specification  process.  Each  subject 
discussed  at  least  four  possible  design  rules. 

Although  these  data  were  not  formally  analyzed,  the  designers  stated  that 
they  utilized  rule-based  behavior;  primarily  for  design  consistency.  From  the 
belief  system  analysis  (Figure  9),  it  would  appear  that  designers  focused  on 
different  areas  of  the  belief  system  matrix  for  rule-based  behavior.  Clearly,  these 
behaviors  that  could  be  described  influenced  the  resulting  designs.  However, 
much  more  investigation  is  necessary  to  determine  the  actual  mapping  of  these 
behaviors  to  the  belief  system  employed. 

The  listing  of  rules  by  subject  below  provides  a  reasonable  start  on 
collecting  design  rules  and  guidelines  for  adaptive  aid  design.  Additionally,  they 
provide  initial  insight  into  how  to  use  design  the  strategies  and  rule-based 
behaviors  to  support  the  design  process. 

C.2  Design  Heuristics 

Subject  1 

1. 

IF:  tasks  to  aid  occur  in  same  basic  context  (i.e.,  similar  task  situation), 

THEN:  attempt  to  aid  consistently  with  previous  tasks. 

2. 

IF:  difficult,  multi-facet  situation  assessment  task 

THEN:  aid  all  parts  that  can  be  automated,  and  leave  most  uncertain  to  pilot 
3. 

IF:  task  is  data  verification  &  validation  or  aircraft  to  aircraft  communication, 
THEN:  automate  it  and  notify  pilot  of  high  priority  information  or  inconsistencies 
in  data  comparison 
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4. 

IF:  novel  foreground  tasks  and  high  workload  and  not  under  attack, 

THEN:  aid  background  tasks 

5. 

IF:  desire  to  aid  tasks  by  type  (e.g.,  situation  assessment) 

THEN:  aid  execution  and  monitoring  first,  then  situation  assessment,  then 
planning  and  committment  type  tasks  (aiding  precedence). 

6. 

IF:  Well  scripted  sequence  of  actions 
THEN:  allocate  to  automation 

7. 

IF:  complex  task  with  scripted,  contingent  decision  making  tasks 
THEN:  partition  according  to  rule  number  2. 

8. 

IF:  beginning  of  plan  execution 

THEN:  transform  displays  to  show  plan  and  estimate  future  status  (momentarily). 
Subject  2 

1. 

IF:  everything  is  going  as  expected 
THEN:  do  nothing 

2. 

IF:  preoccupying  events  occur, 

THEN:  aid  background  tasks  according  to  change  in  performance 
3. 

IF:  difficult  situation  assessment  task 
THEN:  try  to  aid  via  partitioning 
ELSE:  transform  displays 
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4. 

IF:  atactic,  etc.  is  well-known  (e.g.,  16  decoys  problem), 

THEN:  have  automated  situation  assessment  configured  to  check  for  pilot 

5. 

IF:  critical  (e.g.,  launch  weapon)  task, 

THEN:  allocate  fully  to  pilot 

6. 

IF:  event  is  something  the  system  definitely  understands, 

THEN:  let  the  event  occurrence  drive  aiding  need. 


Subject  3 

1. 

IF:  just  beginning  the  specification  process, 

THEN:  do  Fitt's  law  (static  allocation)  analysis  first. 

2. 

IF:  specifying  an  cognitively  complex  task, 

THEN:  look  to  partition  task  first. 

3. 

IF:  specifying  aiding  information  output, 

THEN:  product  information  priority  is  always  high. 

4. 

IF:  the  source  of  the  information  used  in  aiding  is  important, 
THEN:  process  information  has  high  priority. 

5. 

IF:  environmental  or  system  status  information  :s  important, 
THEN:  procedural  information  has  high  priority. 
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6. 

IF:  true  (i.e.,  dynamic)  adaptive  aiding  is  specified, 

THEN:  there  will  be  a  shift  in  the  design  model  coefficients  according  to  the 
dynamics. 


Subject  4 

1. 

IF:  specifying  aiding  for  a  sensor  fusion  task, 

THEN:  allocate  task  to  aiding. 

2. 

IF:  critical  decision, 

THEN:  allocate  decision  to  pilot. 

3. 

IF:  high  level  decision  alternatives  being  evaluated, 
THEN:  allocate  to  pilot. 

4. 

IF:  aiding  produces  alternatives  for  pilot  to  evaluate, 
THEN:  process  information  has  high  priority. 
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